New Hubitat to InfluxDB Bridge via MakerAPI+Node [beta]

2.1.6 Added a nice improvement to the MakerAPI which brought in an official push method, vs WebSocket. Since people have always worried about the InfluxDB Logger user app causing hub slowdowns*, I thought this would be an interesting approach to send data to InfluxDB since it avoids using a User App on the Hubitat.

Code: hubitat-maker-to-influxdb - npm

Warning: This code is very alpha, I've been running it for a bit, but it's not even my main feed of InfluxDB graph data. I'd love to get feedback from others on how it works for them and what problems they see.

What this allows:

Credit (I need to add this in app.js):

  • InfluxDBLogger user app for how to transform event data for InfluxDB. Converting Groovy to JS is amazingly easy.
  • @dan.t Scanned his code for using urlencode for encoding the request to Hubitat.

What you need:

  1. External computer, for example a raspberry pi.
  2. An influxdb server setup on said computer, bound to 127.0.0.1 without auth.
  3. Grafana installed to graph the data.

What it doesn't do that it should:

  1. It probably crashes on unexpected/poorly formed input (send me what the data looks like)

How to use it:

  1. On the Hubitat install a new instance of the MakerAPI. You CAN NOT use an existing MakerAPI that's being used for push, for example the awesome homebridge-hubitat-maker nodejs app from @dan.t
  2. Select devices you want to log data from. Ex: Motion Sensors that send temperature data, power or energy sensors
  3. Install hubitat-maker-to-influxdb from npm
  4. Create a config.json described below. Multiple hubs, or maker apis are supported. Create a new hub for each hub/maker ip. You can send each to a different InfluxDB database.
  5. Run hubitat-maker-to-influxdb, watch the logs.

Place a config file in: ~/.hubitat-maker-to-influxdb/config.json

config.json example is:

{
    "local_config" : {
        "local_url": "http://<ip address of nodejs server>",
        "influxdb_port": 8086,
        "influxdb_host": "127.0.0.1",
        "influxdb_db_name": "hubitatMaker"
    },
    "hubs" : {
        "homeTester": {
            "url": "http://<hub-ip>/apps/api/<maker-app-number>",
            "token":  "<ACCESS_TOKEN>",
            "influxdb_db_name": "HubitatTester"
        },
        "home": {
            "url": "http://192.168.7.97/apps/api/769",
            "token": "abcd-12345-etc",
            "locationName": "Home",
            "locationId": "0",
            "hubId": "0"
        }
    }
}

0.1.7 released

I've updated the npm version to 0.1.7 this adds:

  1. Repeats the last value every 30m if there's no new events. This helps when you zoom in on grafana for devices that don't generate many events.
  2. Added metrics on events and measurements that update every 5 minutes.
  3. Some bug fixes, and support hsm and sunsire/sunset events.

New Metrics about nodejs app:

  • measurementEvent: Number of events of a certain type for the 15 minute interval. For example for 1 power (watts) report from a Zooz meter, and 2 power reports from a Zigbee power meter would mean there were 3 power events.
  • deviceEvent: Number of events of all types for deviceName. For example if your Zooz Power meter sends 2 power events and 1 energy events deviceEvent for it would be 3.
  • measurementDeviceEvent: Number of measurements types for a single devices. For example 2 power reports from your Zooz power meter would be 2 power Zooz.
  • appEvents: 3 different metrics, totalEvents is the count of all events from all devices. influxDBPosts is the number of calls to influxdb, this should be less than the totalEvents, since multple hub events are combined into 1 influxdb call. influxDBErrors is the number of events sent to influxdb that were not imported properly and generated an error.

Why?

Both measurementEvents and deviceEvents are good to find out if you have a device spamming your hub. This is pretty easy with some power meters when set to very low notification thresholds. I can see hub latency increase/be more variable when I had a power meter on a roomba reporting updates every few seconds. Turns out the roomba switches between ~0 watts and 20watts every couple of seconds. Is this important?

*) Though I think recent changes have surely solved this potential problem by bulk sending InfluxDB events.

11 Likes

@bravenel Sorry to tag you directly, but maybe the above post will give context?

Request: Can the Maker API return unit for devices value?

User apps can get this, the maker api returns the unit in post events, but when I get device attributes/events from the maker api, I only get value, but no unit.

Why: When graphing data from influxdb it's nice to have a minimum interval to data or else graphs go blank as you zoom in. For example 1 point/hour would be nice. I can work around this and repeat the last event, but on app restart, new devices, etc, etc I don't know the unit of a value until the devices produces an event if ever. This makes an inconsistent experience for users, and a pain for the maker api user. (plus from a data stand point, it would be nice to now the unit of the data anyways right?)

Example:

Post example:

{ name: 'power',
  value: '22.9',
  displayName: 'Roomba Back House Switch',
  deviceId: '644',
  descriptionText: 'Roomba Back House Switch power is 22.9W',
  unit: 'W',
  data: null }

But if I use the maker api to get device info, attributes lacks the This text will be hiddenunit:

{
  "id": "644",
  "name": "GE/Jasco Smart Switch",
  "label": "Roomba Back House Switch",
  "attributes": [
    {
      "name": "power",
      "currentValue": 22.9,
      "dataType": "NUMBER"
    },
...

[device]/events:

  {
    "device_id": "644",
    "label": "Roomba Back House Switch",
    "name": "power",
    "value": "23.9",
    "date": "2019-11-09T20:28:24+0000",
    "isStateChange": null,
    "source": "DEVICE"
  }

From the device event view, the unit appears to be stored:

We have added unit to Maker API for its event stream that it posts. This was released in 2.1.6.113. Is this what you are talking about?

1 Like

Yes, but can you also add it to Get Device Info and/or Event History?

OK, I will look into it. Should be good for Event History. I don't think it has meaning in the context of device info, as this is a property of an event.

Unit for event history will be in the next release.

2 Likes

@bravenel Sweet! You're awesome, and that makes sense.

I've not used node much - how do I "run hubitat-maker-to-influxdb"? I can see the folder in /home/pi/node_modules

edit: Ah, cd to ~/node_modules/hubitat-maker-to-influxdb and run 'node app.js'

I get:
pi@raspberrypi:~/node_modules/hubitat-maker-to-influxdb $ node app.js
Loading config: /home/pi/.hubitat-maker-to-influxdb/config.json
Hub: home { url: 'http://192.168.0.120/apps/api/353',
token: 'ea7d8805-8c66-4e6d-b133-2f15947b980b',
locationName: 'Home',
locationId: '0',
hubId: '0' }
Config: { local_config:
{ hostname: '0.0.0.0',
base_port: 8567,
influxdb_port: 8086,
influxdb_host: '127.0.0.1',
influxdb_db_name: 'hubitathome',
local_url: 'http://127.0.0.1' },
hubs:
{ home:
{ url: 'http://192.168.0.120/apps/api/353',
token: 'ea7d8805-8c66-4e6d-b133-2f15947b980b',
locationName: 'Home',
locationId: '0',
hubId: '0' } } }
Creating instance for home port: 8567
home local_url http://127.0.0.1:8567/
home post_to http://192.168.0.120/apps/api/353/postURL/http%3A%2F%2F127.0.0.1%3A8567%2F?access_token=ea7d8805-8c66-4e6d-b133-2f15947b980b
home Server running at http://0.0.0.0:undefined/
home Local url http://127.0.0.1:8567/
home Local url encoded http%3A%2F%2F127.0.0.1%3A8567%2F
home Sending post to: http://192.168.0.120/apps/api/353/postURL/http%3A%2F%2F127.0.0.1%3A8567%2F?access_token=ea7d8805-8c66-4e6d-b133-2f15947b980b
home statusCode: 200
home body: {"url":"http://127.0.0.1:8567/"}

How do I check if it's putting anything into influxdb? I'm familiar with SQL etc, and have found the influx CLI.

Ah cool, you can also just do node hubitat-maker-to-influxdb and it'll run the module/app. I recommend installing pm2, it can demonize and run the app in the background. Quick google will help you there.

So, it's running, it's fairly verbose as it runs. It needs events from Hubitat to send data to influxdb. So go to the Maker API app on the hub and make sure some devices are selected that send data. Worst case, try a twice you can turn on/off.

For each event you should then see it print/log:

home data: power,deviceId=76,deviceName=Dryer\ Switch,hubName=home,hubId=0,locationId=0,locationName=Home,repeat=true,unit=W value=0.000 1573368135898

This is actually the influxdb message it's sending. It batches events and sends them every 5s or so. From the influx cli you can then do select * from <db> if I remember correctly and it'll show you the event.

Hope that helps!

It's not logged anything since the last stuff I posted. I've added a light switch in addition to the temperature sensors and thermostat I added yesterday. I've toggled the light switch but still nothing.

Why does my log say "home Server running at http://0.0.0.0:undefined"? Just so I'm clear, this node.js app does GETs or POSTs to the maker API app running on my HE at 192.168.0.120, and then GETS or POSTs that data (transformed) to influxdb to insert the data? What is the base_port config value of 8567 for?

Thanks for you patience @pjam73

Hmm, can you send me 2 things via PM?

  1. The node logs on startup
  2. A screen shot of the hubitat Maker API app section, URL to send events to by POST should look like:

image

You should see log messages in near real time.

The app does:

  1. A single GET to the hubitat to set a "target" URL that the hubitat will post to. It does it in startup. The target URL is a pointer back to the app itself.
  2. Listens for POSTS from the hubitat.
  3. On events the hubitat calls the url from #1 with a POST.

The URL that's sent to the hubitat is http://<local_config.local_url>:<local_config.base_port>/

base_port is the port in the node application will use to listen for POSTs from the hubitat. It can be anything > 1024 that's not in use. If there's more than 1 hub it increments base_port by 1 and listens on multiple ports so it can tell the hubs apart. (hence why it's base, as opposed to just port)

Ah the undefined is a bug in the log message, it had a typo. I've published 0.1.5, you can update it by same command used to install it. npm i hubitat-maker-to-influxdb

So here's what the config options mean in case it helps:

{
    "local_config" : {
        "local_url": "http://192.168.7.94", -- The URL (http:// + IP address) of the _node_ server. This is really important. NOT hubitat.
        "hostname":  "0.0.0.0",  --- the IP/interface to listen on, 0.0.0.0 is all interfaces
        "base_port": 8567,       --- the port to listen on for the local http server
        "influxdb_port": 8086,   --- The port to find influxdb listening on
        "influxdb_host": "127.0.0.1", --- The ip address for influxdb
        "influxdb_db_name": "hubitatMaker" -- The default database to store data in
    },
    "hubs": {
        "home": {
            "url": "http://192.168.7.97/apps/api/769", -- URL for the maker API on the hubitat.
            "token": "<ACCESS_TOKEN>", -- Auth token for above
            "locationName": "Home",    -- location name, mimics InfluxDB logger Hubitat app (optional)
            "locationId": "0",         -- ditto (optional)
            "hubId": "0"               -- ditto (optional)
        }
    }
}
1 Like

I’m not at home now, but I think the “URL to send POST events to” setting on the hub is the problem. I noticed it was 127.0.0.1 but thought maybe it wasn’t used, clearly it is.

Thank you - it's working now.

Any suggestions as to where to look to find how best to plot multiple device temperatures on a grafana dashboard? e.g. a single graph with a series of data points for each device?

Yay! Glad it's working.

Like this?

Under edit graph (or when creating one) click

That help?

1 Like

or simply

image

will display all your humidity sensors on the one chart. Edit humidity for whatever measurement you desire to chart.

2 Likes

0.1.7 released

I've updated the npm version to 0.1.7 this adds:

  1. Repeats the last value every 30m if there's no new events. This helps when you zoom in on grafana for devices that don't generate many events.
  2. Added metrics on events and measurements that update every 5 minutes.
  3. Some bug fixes, and support hsm and sunsire/sunset events.

New Metrics about nodejs app operation:

  • measurementEvent: Number of events of a certain type for the 15 minute interval. For example for 1 power (watts) report from a Zooz meter, and 2 power reports from a Zigbee power meter would mean there were 3 power events.
  • deviceEvent: Number of events of all types for deviceName. For example if your Zooz Power meter sends 2 power events and 1 energy events deviceEvent for it would be 3.
  • measurementDeviceEvent: Number of measurements types for a single devices. For example 2 power reports from your Zooz power meter would be 2 power Zooz.
  • appEvents: 3 different metrics, totalEvents is the count of all events from all devices. influxDBPosts is the number of calls to influxdb, this should be less than the totalEvents, since multple hub events are combined into 1 influxdb call. influxDBErrors is the number of events sent to influxdb that were not imported properly and generated an error.

Why?

Both measurementEvents and deviceEvents are good to find out if you have a device spamming your hub. This is pretty easy with some power meters when set to very low notification thresholds. I can see hub latency increase/be more variable when I had a power meter on a roomba reporting updates every few seconds. Turns out the roomba switches between ~0 watts and 20watts every couple of seconds. Is this important?

Wow, this was really easy to set up. Great work!

I created a docker-compose file, might be usefule for someone:

services:
  node:
    image: node
    volumes:
     - ./node_modules:/node_modules
     - ./config:/root/.hubitat-maker-to-influxdb/
    ports:
     - 8567:8567
    links:
     - influxdb
    command: "node /node_modules/hubitat-maker-to-influxdb/app.js"
  influxdb:
    image: influxdb
    volumes:
     - ./influxdb:/var/lib/influxdb
    environment:
     - INFLUXDB_DB=hubitatMaker
  grafana:
    image: grafana/grafana
    ports:
     - 3000:3000
    links:
     - influxdb
    environment:
     - GF_SECURITY_ADMIN_PASSWORD=<password>
  1. Create the docker-compose.yml file
  2. Run docker-compose run node bash
  3. Run npm i hubitat-maker-to-influxdb
  4. exit container shell
  5. Customize config.json and place it in config folder ("influxdb_host": "influxdb")
  6. Run docker-compose up -d

This is a quick & dirty approach - of course you could also build a custom node container image like so - Dockerfile:

FROM node
RUN npm i hubitat-maker-to-influxdb
CMD ["node", "/node_modules/hubitat-maker-to-influxdb/app.js"]
EXPOSE 8567
7 Likes

@janwerner Sweet! that's really awesome

1 Like

Hello, thank you for this node js app and all the info, I really appreciate it! It works for me but randomly I get this on some of my devices... switches, dimmers, ecobee, etc...

Here is an example of an entry way inovelli red series dimmer event:

home STATUS: 400

home HEADERS: {"content-type":"application/json","request-id":"2ad8597e-542b-11ea-804c-00155d48df0a","x-influxdb-build":"OSS","x-influxdb-error":"partial write: unable to parse 'lastActivity,deviceId=106,deviceName=Entry\\ lights,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false,unit= Feb  value=\"2020\" 1582235491731': missing tag value dropped=0","x-influxdb-version":"1.7.10","x-request-id":"2ad8597e-542b-11ea-804c-00155d48df0a","date":"Thu, 20 Feb 2020 21:51:36 GMT","content-length":"236","connection":"close"}

home BODY: {"error":"partial write: unable to parse 'lastActivity,deviceId=106,deviceName=Entry\\ lights,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false,unit= Feb  value=\"2020\" 1582235491731': missing tag value dropped=0"}

Not sure what missing tag value dropped=0 means. Any insight on this would be much appreciated.

Thanks!

@Dolby Sorry for the delay!

Off that's annoying. The driver for now has set the unit to " Feb" (see the space) which of course fails to be read by the database. It's not the end of the world, it'll load the rest of the data. Annoying attribute, lastActivity sends the month name? erg.

Drivers can make up any attribute they want, and this is messed up. I'm pretty sure there's no defined lastActivity attribute, and certain it shouldn't have a space.

2 solutions:

  1. Really should fix the driver. It's generating bad events. Is this a community driver?

  2. I'll strip whitespace around units/values/etc before posting them to the DB. Give me a couple of days to do this, I've got a few other projects going on taking my time.

Thanks @asj! Take your time this is not anything important. I really appreciate you taking a look at it and providing info on the issue I was having. I'm not that familiar with npm but if I figure out how to escape spaces in the js I'll open a PR.

The drivers used for the entry lights are from Inovelli's repo GitHub - InovelliUSA/Hubitat: Location for Inovelli Hubitat Drivers and Apps

I see there is an update to the drivers I use from the Inovelli repo and I'll see if that fixes the whitespace issue or maybe I can identify the offending code and fix it in their driver.

I would use the hubitat driver for the Inovelli dimmers/switches however its missing some features at the moment.

The built in ecobee driver seems to have a similar issue...

home STATUS: 400

home HEADERS: {"content-type":"application/json","request-id":"03cfff10-5751-11ea-84fa-00155d48df0a","x-influxdb-build":"OSS","x-influxdb-error":"partial write: unable to parse 'thermostat,deviceId=98,deviceName=Main\\ Floor,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false value=heating 1582581600621': invalid boolean dropped=0","x-influxdb-version":"1.7.10","x-request-id":"03cfff10-5751-11ea-84fa-00155d48df0a","date":"Mon, 24 Feb 2020 22:00:05 GMT","content-length":"217","connection":"close"}

home BODY: {"error":"partial write: unable to parse 'thermostat,deviceId=98,deviceName=Main\\ Floor,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false value=heating 1582581600621': invalid boolean dropped=0"}

This is another one... Seems to be related to the furnace running or idle status?

home STATUS: 400

home HEADERS: {"content-type":"application/json","request-id":"cb2aa920-575a-11ea-850a-00155d48df0a","x-influxdb-build":"OSS","x-influxdb-error":"partial write: unable to parse 'thermostat,deviceId=98,deviceName=Main\\ Floor,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false value=idle 1582585800541': invalid boolean dropped=0","x-influxdb-version":"1.7.10","x-request-id":"cb2aa920-575a-11ea-850a-00155d48df0a","date":"Mon, 24 Feb 2020 23:10:05 GMT","content-length":"214","connection":"close"}

home BODY: {"error":"partial write: unable to parse 'thermostat,deviceId=98,deviceName=Main\\ Floor,hubName=home,hubId=0,locationId=0,locationName=Waubeesee,repeat=false value=idle 1582585800541': invalid boolean dropped=0"}