Maker API: Device Attributes

[zbdrv]

Hi,

So I'm working on a project that makes use of the Maker API to scrape data from sensors so I can graph the data. This is mostly fine and a lot of the data presented is self explanatory, or has a decent enough explanation at docs.hubitat.com.

However, the attributes map returned for devices seems a little strange. The dataType and values keys don't seem to mean anything.

For example, here we have a light bulb:

    "attributes": {
      "colorTemperature": null,
      "dataType": "ENUM",
      "values": [
        "on",
        "off"
      ],
      "level": "100",
      "switch": "off",
      "colorName": null
    },

OK, great, the dataType seems to tell us that values is an ENUM and those values are clearly the values that switch can be. But there are also colorName, colorTemperature, and level, why no information on what those could be?

Next, we look at an outlet:

    "attributes": {
      "energy": "0.134",
      "dataType": "NUMBER",
      "values": null,
      "power": "0",
      "switch": "off",
      "checkInterval": "720"
    },

This time dataType is NUMBER and values is null. This appears to be useless. We have a lot of number values here, which was it talking about? And why is the switch ENUM no longer mentioned anywhere?

Off we go to the docs looking for an explanation of this and we find the Attribute Object, but there's no real explanation for what the Maker API is returning.

It feels like the attributes map in its current form may be merging a bunch of other attribute maps together to generate its content, meaning we end up with the values and dataType from one of our attributes, but we don't know which, making them essentially random and useless.

Would the attributes map perhaps be improved if it was exposed as:

"attributes": {
  "switch": {
    "dataType": "ENUM",
    "values": [
      "on",
      "off"
    ],
    "value": "on"
  },
  "energy": {
    "dataType": "NUMBER",
    "values": null,
    "value": "0.134"
  }
}

This tells you the precise dataType, values (if possible), and value for each attribute in an unambiguous way.

I may also be looking at this completely incorrectly and there is some logic behind which attribute dataType and values correspond to.

So, does anyone know how those values work, or know which part of the documentation I've missed?

Thanks,

[JasonJoel]

Which endpoint are you using?

For instance, I've seen cases where devices/all yields different info than devices/* (I never use devices/all because of that in favor of devices/*).... So we may need to be very specific here.

The format you posted above looks suspiciously like the result from devices/all, which again does weird crap and shouldn't be used (in my opinion - that's not the official Hubitat stance).

[zbdrv]

Which endpoint are you using?

You're correct, this is the output from devices/all. I've just looked at the devices/{id} endpoint, and that looks much more reasonable. I think it confirms what I suspected was going on with the devices/all endpoint too, the attribute maps are being merged in some strange way, mangling the output and making the dataType and values fields nonsense (perhaps they should just be omitted to avoid confusion).

Thanks for pointing this out, I should have checked the device specific endpoints before asking here, but I thought devices/all would be reasonable.

I'll use the devices endpoint to enumerate devices and the devices/{id} specific endpoints for grabbing the actual data.

Thanks.

[JasonJoel]

Anywhere you were using "devices/all" consider using "devices/*". That will return one large response for all devices, like devices/all did, but in the same formal as devices/(device id).

[zbdrv]

devices/*

Ahh, I didn't realise you meant that URL literally. Yes, this output looks very good and includes everything. I'm surprised this isn't on the Maker API page instead of devices/all.

Thanks again

[JasonJoel]

@zbdrv

I'm surprised this isn't on the Maker API page instead of devices/all.

Well, I've suggested that before.... Hubitat is very good about not removing functionality and breaking things for end users - and there are a number of solutions that use "devices/all" still.

@bravenel I do think that devices/* should be documented better/suggested over devices/all though. The mangled json response in some parts of devices/all keeps coming up, and is totally avoided when using devices/*...

[zbdrv]

Looking further into devices/*, even this output is kind of bizarre.

    "capabilities": [
      "Configuration",
      "Actuator",
      "Refresh",
      "PowerMeter",
      {
        "attributes": [
          {
            "name": "power",
            "dataType": null
          }
        ]
      },
      "EnergyMeter",
      {
        "attributes": [
          {
            "name": "energy",
            "dataType": null
          }
        ]
      },
      "Switch",
      {
        "attributes": [
          {
            "name": "switch",
            "dataType": null
          }
        ]
      },
      "Sensor",
      "HealthCheck",
      {
        "attributes": [
          {
            "name": "checkInterval",
            "dataType": null
          }
        ]
      }
    ],

You've got random maps just thrown into the middle of the capabilities array. Well, they aren't random, they're saying something about the capability that preceeded them, but this feels like an awful way of doing things.

A better way of presenting which capability an attribute came from would probably be to mention the capability in the attributes data, instead of completely butchering this capabilities array.

eg.

  "attributes": [
    {
      "name": "colorTemperature",
      "currentValue": null,
      "dataType": "NUMBER",
      "capability": "ColorTemperature"
    }
  ]

As it is now, you can't meaningfully parse capabilities without looking ahead to see what the next element in the array is, making things very complicated.
Is the next element a string, meaning it's just a capability name, or is it a map, meaning there's some meaningful data about the thing I just parsed?

If this data were to remain in the capabilities array, it should be changed to something better. A capabilities map, for example, where the parsing would be predictable and the attributes would be attached to the capability name instead of just inserted as the next element in the array.

"capabilities": {
  "ColorTemperature": {
    "attributes": [
      {
        "name": "colorTemperature",
        "etc": "andSoOn"
      }
    ]
  },
  "Refresh": {
    "attributes": [
    ]
  },
  "Etc": {
  }
}

[bravenel]

@zbdrv

A better way of presenting which capability an attribute came from would probably be

At this point I am disinclined to touch these details of Maker API, as there is no way to know the impact of doing so on existing installations.

@JasonJoel

I do think that devices/* should be documented better/suggested over devices/all though.

I will pass this on to the documentation team...

[Angus_M]

/* is also a heck of a lot faster than /all.

[JasonJoel]

It is, although less than it used to be.

/* used to return data 3-4x faster than devices/all, but presumably through other platform work the two seem to be a lot closer in speed than they used to be (devices/all got faster) - at least on my hubs.

[zbdrv]

@bravenel

At this point I am disinclined to touch these details of Maker API, as there is no way to know the impact of doing so on existing installations.

An understandable predicament. Perhaps a nice workaround for that would be to version the API, so breaking changes can happen and newer data can be presented under a new endpoint without upsetting applications using the older endpoints. This would also allow you to nicely deprecate and eventually delete older non-supported endpoints if you so choose.

If I'm unable to work out a decent way of parsing the Maker API capabilities, is it possible for me to write an app for the HE itself, and have that app reachable over an HTTP endpoint like the Maker API? If so, I can write something that returns a set of data like devices/*, but easier to parse.

It looks like all of the pieces are there at docs.hubitat.com, I'd just like to ensure that creating endpoints for our own apps is possible.

Thanks.

[bravenel]

What you suggest may be possible.

But I don't understand your problem. Are you not able to parse the JSON returned into a map? You said something about "there's some meaningful data about the thing I just parsed?", which implies something sequential about your parsing.

There is not a one-to-one correspondence between attributes and capabilities. Each device capability has associated attributes with possible values, and these are returned in the capability section of the object. The attributes of the device include those and possibly others with possible values, and these are returned in the attributes section of the object. It should not matter the order these two sections appear in the JSON.

[zbdrv]

Are you not able to parse the JSON returned into a map?

Probably, but it's quite difficult. I'm Parsing this in Rust using the serde library. To parse the capabilities array, I'd have to use a custom Deserializer due to what's being done there.

If it were an array of capability names (strings), it could be parsed as Vec<String>, if it were an array of maps, it could be parsed as Vec<HashMap<String, serde_json::Value>>, etc. But capabilities is instead a strange array with mixed strings and maps that isn't directly parsable as a map.

You said something about "there's some meaningful data about the thing I just parsed?", which implies something sequential about your parsing.

I'm implying that the capabilities array is kind of broken. See this snippet of my above capabilities array:

  "capabilities": [
      "Switch",
      {
        "attributes": [
          {
            "name": "switch",
            "dataType": null
          }
        ]
      },
      "Sensor",
      "HealthCheck",
      {
        "attributes": [
          {
            "name": "checkInterval",
            "dataType": null
          }
        ]
      }
  ]

We get the first element. Switch, the next element of the array is a map with some attributes that are meaningful to Switch.
We get the second element, Sensor, the next element is another capability. Nothing to do with Sensor.
We get the third element, HealthCheck, the next element of the array is a map with some attributes that are meaningful to HealthCheck.

The parsing MUST look ahead somehow if it's to present something useful at the end of the parse, because you MUST discover if the next array element is your attributes or not.

[bravenel]

@zbdrv

Probably, but it's quite difficult. I'm Parsing this in Rust using the serde library.

That's unfortunate. It hardly seems like a mainstream use case for dealing with JSON objects. By definition, JSON objects are objects, not streams of characters.

@zbdrv

is it possible for me to write an app for the HE itself, and have that app reachable over an HTTP endpoint like the Maker API? If so, I can write something that returns a set of data like devices/*, but easier to parse.

You could certainly write an app for the hub that loads the JSON from Maker API (via an endpoint call) and formats it however you need it formatted. It is very easy to parse the JSON into a map in Groovy. Supporting endpoints is also quite easy for an app on the hub.

[zbdrv]

That's unfortunate. It hardly seems like a mainstream use case for dealing with JSON objects. By definition, JSON objects are objects, not streams of characters.

I'm not quite sure what's being said here, but it's possibly because I'm not communicating well enough.

The serde library for Rust is excellent. I can very easily quickly parse arbitrary JSON into a map or struct, that's not an issue.

The issue is that to turn the capabilities array into a map at parse time (I took "Are you not able to parse the JSON returned into a map?" to mean specifically the capabilities array, not the whole devices/* return) would require the deserializer to have specialised knowledge of this array. To turn the array into a map later on requires a second pass over the array to produce a map.

This is a fault of the capabilities array and not the language chosen to parse the array in, or the library chosen to perform the parsing.

I think the simplest I can put it is this: If you encounter a map in the capabilities array, you MUST at some point look at the previous element to give meaning to that map. This doesn't seem like a great way to convey the data when the API could just present it in a straight forward way.

For now, I can actually just ignore the capabilities array, it doesn't really contain any data I need for what I want to do. It's just a shame that that data is more complicated to use than it needs to be.

I also apologise if I sound adversarial here, I don't mean to. I'm just having a hard time communicating the problems with the capabilties array.

Thanks for engaging with me here.

[bravenel]

The list of capabilities, which I assume you are calling the "capabilities array", is a list where the first element is the name of the capability, and the next element is a map of the attributes for that capability, like this:

[capab-1-name, ["attributes": capab-1-attributes], 
 capab-2-name, ["attributes": capab-2-attributes], 
 ...]

If what you want is for the capabilities list to instead be a list of maps, like this:

[["name": capab-1-name, "attributes": capab-1-attributes],
 ["name": capab-2-name, "attributes": capab-2-attributes],
 ...]

That could be done by creating a new endpoint in Maker API.

[zbdrv]

@bravenel

The list of capabilities, which I assume you are calling the "capabilities array"

Correct, I'm referring to the capabilities list, sorry if that caused confusion.

@bravenel

is a list where the first element is the name of the capability, and the next element is a map of the attributes for that capability, each , like this:

[capab-1-name, ["attributes": capab-1-attributes], 
 capab-2-name, ["attributes": capab-2-attributes], 
 ...]

Unfortunately this is false. If that were the case, parsing it might be quite a bit easier.

Here is a snippet from devices/* on a random device from my HE right now:

    "capabilities": [
      "Actuator",
      "Refresh",
      "ColorTemperature",
      {
        "attributes": [
          {
            "name": "colorTemperature",
            "dataType": null
          },
          {
            "name": "colorName",
            "dataType": null
          }
        ]
      },
      "ChangeLevel",
      "SwitchLevel",
      {
        "attributes": [
          {
            "name": "level",
            "dataType": null
          }
        ]
      }
    ]

Here we see it's not alternating as [string, map, string, map] as you mentioned, it's going: [string, string, string, map, string, string, map]. It appears to be omitting the map when (presumably) there are no attributes for the capability. This is why parsing it is difficult.

@bravenel

If what you want is for the capabilities list to instead be a list of maps

I believe that would certainly be easier to parse, as long as the attributes map weren't omitted if there were no attributes as it is now. It should just be null or an empty list.

[bravenel]

if (attribs) caps << ["attributes": attribs]

Yeah, it simply omits the map of attributes when there are none.

So the first element of the capabilities list is a string. The next element is either a map or another string. Etc. I'm not seeing why that is so difficult to deal with.

[zbdrv]

I'm not seeing why that is so difficult to deal with.

It does appear that I am unable to sufficiently convey that this data structure isn't great, and that it is difficult to work with. I believe anything I'd say now would just be reiterating issues I mentioned earlier in the thread, so I think I'll just have to leave it here and attempt to create some workarounds or just ignore the capabilities list completely.

Thanks for your time.

[bravenel]

Yeah, I understand. The app's been this way for 3 years, and you're the first person to raise this issue. We have a lot of users with Maker API in production. I have no pride of authorship here as I didn't write it, and can't speak for the person who did as to why it is the way it is. And, it's not clear to me what the whole capabilities section brings to the party for most use cases.