InfluxDB Logger Support

Hi all,

There have been a number of changes to the InfluxDB logger recently. Here is a summary of 2.3/2.4:

  • Cleanup and rationalization of the UI. Hopefully this will make things much easier to see/find/change in the configuration.
  • The write batching to InfluxDB has been completely redone. Batches are now based on elapsed time since first event of the batch rather than a periodic timer.
  • Softpoll (aka keep-alive) now only generates events for devices that have not already produced an event within the last interval. If you are using keep-alive, this should greatly reduce/remove unnecessary events.
  • Retry failed posts to InfluxDB. If the post to InfluxDB fails, events are now retained and included in the next batch posted to InfluxDB. This addresses issues of data loss when the InfluxDB server is unavailable.
  • Allow Hub Name and Location tags to be disabled for device events. The automatic inclusion of Hub Name and Hub Location in device events can now be disabled, which can significantly reduce the size of events. Unless you have multiple hubs with conflicting device names (such as two hubs both with a device called "Hallway light"), you might want to consider this option.
  • Improved logging. The logging has been improved and made more functional for "everyday use."

The current version, 2.4.1, is available via HPM or directly from the repo.

Please let me know if you have any questions.

3 Likes

It seems like over the last day or so if any of my data has an issue it causes all of the data to not upload. Right now It's about 2 hours behind when I look in Grafana. I do see the logs:

Request, Headers: [X-Influxdb-Error:partial write: field type conflict: input field "value" on measurement "level" is type string, already exists as type float dropped=1, X-Request-Id:d25ac06f-b961-11ed-a8a9-0242ac110004, X-Influxdb-Build:OSS, X-Influxdb-Version:1

So I know it doesn't like the data from 2 devices but should that cause all the data to fail and have to retry? I am only asking because in the last update I saw Retry failed posts and maybe there is an issue even if the server is available but the data for a couple devices fail because of a field mismatch.

Thank you

It's receiving an HTTP post failure from InfluxDB--a return code at or above 400.

Three things:

  • What version of InfluxDB are you using?
  • Can you tell me what the return code is please? It should be an error in your Habitat log
  • Can you tell me about the two devices? What kind they are?

Thanks

I am using InfluxDB 1.8.9

The return code is 400, here is the full output:
2023-03-03 07:36:39.613 AM[error] Post of 50 events failed - elapsed time 0.716 seconds - Status: 400, Error: Bad Request, Headers: [X-Influxdb-Error:partial write: field type conflict: input field "value" on measurement "level" is type string, already exists as type float dropped=1, X-Request-Id:06e94b91-b9c0-11ed-ba17-0242ac110004, X-Influxdb-Build:OSS, X-Influxdb-Version:1.8.9, Content-Length:150, Request-Id:06e94b91-b9c0-11ed-ba17-0242ac110004, Date:Fri, 03 Mar 2023 12:36:32 GMT, Content-Type:application/json], Data: null

The 2 devices are Flair smart vents which are cloud based and it seems when the level is 0 in the logs it shows: value="\ 0" so I am thinking it's the \ in there it doesn't like. I had other errors yesterday with my Switchbot curtain because the level was set to "Unknown" and it didn't like that.

Eventually I will see in the logs that it drops data because it passes the queue level:

2023-03-03 07:50:09.661 AM[debug] Post of size 50 events already running (elapsed 5092ms)

2023-03-03 07:50:09.659 AM[debug] Number of events queued for InfluxDB: 4830

2023-03-03 07:50:09.657 AM[debug] Maximum queue size reached

I am fairly new to using InfluxDB so my issue could be something outside of your smart app but I have been running it for a few months and only ran into this after the last update so I thought I would start here.

Thank you,
Ian

Can you clarify something. Are you using the option for "Acess to all Attributes". Also what is the attribute that is being sent with this.

400 Generally means that the data being sent in the call is malformed. So it is basically bad data that Influx DB can't load. This will happen can easily happen if the data has special characters in it for some reason.

Or are these devices selected by using the regular Switch Level cabaility?

I am using the Advanced Attribute Selection and I have "level" selected for the device and the driver is using capability "Switch Level". If I go into the events for the Vent It just shows 0:
Name Value Unit
level 0 %

When I enable Debug logging though I see :
level,deviceId=xxx,deviceName=xxx,hubName=xxx,locationName=xxx,unit=% value="\ 0" 1677760651635000000

I am assuming that because the value is "\ 0" and not "0" that is why I am getting the error. I am not sure where the \ is coming from. Even if only 2 values fail though should it prevent all of the data from being sent to influxDB?

Thank you,
Ian

Switch Level is a standard attribute and as such shouldn't be a issue. It seems that for some reason there is space before the 0 coming from the driver in the Hubitat, and so when it goes through the influxDB Logger it is catching that space and adding a \ before it. This is needed for INfluxDB Logger to handle spaces in data that are strings. I wonder if influxDb is expecting that value to be numerical instead of character which is what would happen with that change.

It breaks the parser so it can't parse the entire set of data, and that is why it isn't getting any data into the database for that post action.

So this is actually expected by the InfluxDB Logger app and more of a problem of bad data in, bad data out. You will need to get someone to take a look at that driver to find out why that space is included and why that value is charecter. It should be numeric.

That makes a lot of sense. After comparing my Keen vent with my Flair vent it looks like when the flair vent is 0 it puts in % under Unit and I wonder if that is causing the space. I don't see % under unit when it's anything other than 0 and my Keen doesn't show anything under Unit for any level. I will do some digging.

Using Webcore to show the level it doesn't add any space, does InfluxDB take the Unit into consideration when displaying the value?

Thank you for pointing me in the right direction!

The back-slash is an escape character. All strings sent to InfluxDB are escaped for special characters. Space is meaningful to the InfluxDB line parser, so the back-slash is added to indicate to InfluxDB that it is not a separator, but a space inside a string. Unclear if the back-slash is actually necessary since the string is in double quotes, but it doesn't hurt.

Short version is that there is a bug in the driver. The level attribute is being set to a string of " 0" (with a leading space) rather than to "0". Perviously what was happening is that InfluxDB would silently drop the entire post (dataloss). Now the app is checking the return and detecting the error so that the post can be retried.

So, first thing to do is to turn off the attribute level for the Flair vents, and ask the driver maintainer to fix the driver. Don't turn it back on again until the driver is fixed.

Second thing. You can address the backlog by grabbing the pre-release version of InfluxDB-logger available here. Note that if you are using HPM, you will need to install the app by hand in Developer Tools -> Apps Code.

One the pre-release is installed, go into the configuration and set "Batch size limit" to 1, and "Backlog size limit" to 1, then save the app. This will cause each event to be individually posted to InfluxDB, and any post errors will cause that single event to be discarded. Expect to see a bunch of warnings in the log from each post and about the amount of data in the backlog, and an error for every piece of bad data.

Once the backlog goes down and the errors have stopped, set your "Batch size limit" and "Backlog size limit" back to normal values.

2 Likes

Followed your steps and I have everything caught up now. I will reach out to the developer of the driver to see what can be done before logging that device again.

Thank you so much for your help!
Ian

1 Like

I added a second instance of the app to use advanced attribute selection for hub info. At first, I selected 3 attributes for temp including C and F. Then I started seeing this in the logs:

So I went back in and changed that attributes to only be:
image
but I still get the warn log entries. How do I clear that?

The warning comes from the value of this custom event, which is a string that contains an unescaped space character between the numerical value and the unit. I think it's there mostly to make the creation of html tile templates more straightforward.

Hmm strange. Maybe double-check your attribute selection (in both instances?) and make sure you hit "Done".

It's driver brokenness. The strings "temperatureC" and "temperatureF" are not valid Hubitat event types. The correct event type would be "temperature", along with unit="°F" or unit="°C".

[Edit: never mind, I see what driver it is]

@JamesTX10

Strike that. My bad. Apologies to everyone, particularly @thebearmay.

I looked at the Hub Information driver, and it is absolutely not broken. I should have looked at it before posting. As noted by @hubitrep, the attributes "temperatureF" and "temperatureC" are formatted for browser presentation and should not be used for InfluxDB loggering. There is a standard temperature attribute which has the value and unit in separate fields as expected.

Remove attributes "temperatureF" and "temperatureC" from InfluxDB-logger, and use attribute "temperature" instead. Alternatively, you can just turn off individual attribute selection and select Hub Information in the Temperature Sensors section. You probably want to delete or convert your prior records for "temperatureF" and "temperatureC" as Grafana is not going to deal with these particularly well.

Sorry I mislead you earlier!

3 Likes

Vision 2.5 has just been pushed. Changes of interest are:

  • Fix button and thermostat fan mode event handling
  • Fix threeAxis event encoding
  • Add device event handling for filters, gas detectors, power source
  • Move unnecessary info messages to debug
  • Disable debug logging of post data which drives hubs into the ground
  • Provide info logging of event data to replace post data logging
  • Allow backlog to be set as low as 1, allowing bad records to be cleared

The last bullet is particularly useful if you should find yourself with bad data in your queue like this. If you end up with bad data in your queue, you can use the following steps to clear it:

  1. Go into the configuration for InfluxDB Logger and remove whatever device and/or attibute is creating the bad data from the Devices To Monitor list.
  2. Set both the "Batch size limit" and "Backlog size limit" to 1.
  3. Save the configuration.
  4. Let the queue process. Depending upon the size of the backlog, this may take a while. Expect to see a bunch of warnings and errors in the log--a warning on every post about the amount of data remaining in the backlog, and an error for every piece of bad data in the backlog.
  5. Once the backlog has been processed and the errors have stopped (the warnings will continue), go back into the configuration and change the "Batch size limit" and "Backlog size limit" back to normal values and save the configuration.

InfluxDB on Synology with HE - Step By Step:

[details="InfluxDB Steps"] UPDATE _ Docker changed to Container Manager Instructions are different as of 10/10/23 **

Access your NAS web screen usually http://{NASName}:5000 or http://xxx.xxx.xxx.xxx:5000
I have to assume you've already created a pool/shared folder for Docker - mine is simply called 'docker'.
Open File Station
Select the storage location you've setup - in my case its SynologyNAS1 'docker'
Select Create folder from the Create dropdown
Enter a folder name - I put a name 'influxdb'. I suggest all lower case - unix case sensitivity etc.
Right click the folder, select Properties and change owner to 'admin' if needed. Also check the 'apply to sub-folders'. Save
Close File Station
Back on main screen select 'Docker'
On left menu select Registry.
on right side, enter 'influxdb' in search and hit enter.
Hilight The influxdb option at the top with the largest quantity of rating stars. This is the latest open source.
Select Download above it.
When prompted for Tag choose 'latest' - then 'Select'.
On the left menu, select Image. You should see the influxdb - if the Disk icon next to its name is changing - it is still downloading. wait for it to complete.
Highlight influxdb:latest and click 'Launch'
The first prompt is networks - select 'same as docker host then Next
Change the Container name to 'influxdb'.
click 'Enable auto-restart'. Then Next. do not be tempted to click the web portal via web station! Click Next
Click 'Add Folder'. In the popup - drill to your folder name created above mine was 'SynologyNAS1 : docker : influxdb' highlight it and click 'Select'.
Enter a mount path - use ' /var/lib/influxdb ' and click 'Next'.
Review and click 'Done'.
On the left menu - select Container - influxdb should say 'running'.
Close Docker

From your browser, access InfluxDB portal - http://{NASName}:8086 or http://xxx.xxx.xxx.xxx:8086
Click 'Getting Started'
on Setup Initial User - put admin (lower case) then a password - remember this info!
confirm the password, and add an Initial Org Name - I put my city. I use an uppercase first later as it is a proper name.
For Initial Bucket Name I put Hubitat. Like City - it is a name, so first character is Uppercase. Click Continue
Select 'Advanced'.
On the Load Data screen - select API Tokens at the top
Select Generate API Token button on the right
Choose All Access API Token
Give the Token a description - I use 'TOKEN_FOR_HE' - COPY THE TOKEN CREATED and save it to a text file or something!

Open your hubitat webpage
On the left - select Apps
On the right - select Hubitat Package Manager
Click Install
Select Search By Keywords
Enter 'Influxdb' for search criteria then select 'Next'.
I get TWO identical responses - I cannot see a difference but I take the second one.
the Install from Repository screen appears - click Next and Next again.
Shortly, the app should be installed - click Next then Done at the bottom to leave HPM app.
At the top of the Hubitat window you've returned to - select 'Add User App'.
Scroll popup to find InfluxDB Logger. Shortly, you should be IN the setup of InfluxDB Logger app
Leave all values alone for now - but select 'InfluxDB Connection' button.
Enter Host* as IP address for the SynologyNAS. Hostname {NASName} requires a DNS entry and HE is bad at DNS names(no host.etc file).
I set log to 'error' initially, then to Info after I verify it works.
Leave the port at 8086
Change Influx Version from 1.x to 2.x
For Org put what you entered in the Influxdb page - remember I had suggested your city.
For Bucket enter 'Hubitat'.
For Authorization Type select Token. Put the Token we created previously and saved - here. Click done
Back on the app main setup page -
Click 'Advanced Attribute selection' - its easier to see your devices.
Pick ONE easy thing to test with.
I selected a Humidity Sensor.
Pick an attribute or two - I chose temperatre, battery, and humidity.
Open Logs - and look for 'InfluxDB Logger: Updated' and 'InfluxDB URI: http://xxx.xxx.xxx.xxx:8086/api/v2/write?org={cityname}&bucket=Hubitat
It could take a few minutes - depending on what your update value in the logger was set to.

Open the InfluxDB webpage. If you had it open from before, be SURE to refresh it!
After logging into Influxdb page, select the 'graph' icon - called Data Explorer from the Left menu
On the right, look for the query1 and below that you should see 'Hubitat' and 2 other buckets that Influx makes automatically.
Select Hubitat.
To the right, a new popup of 'Filter' appears. It should show you the attributes you had previously selected on the HE Influx DB Logger app.
Check mark one of them and over to the left select 'Submit'.
Viola! You should see the beginnings of a graph from the incoming data. Have fun.

[/details]
Grafana step by step - includes anonymous access, and add to web tile!

Grafana steps

Access nas web screen usually http://{NASName}:5000 or http://xxx.xxx.xxx.xxx:5000
I have to assume you've already created a pool/shared folder for Docker - mine is simply called 'docker'.
Open File Station
Select the storage location you've setup - in my case its SynologyNAS1 'docker'
Select Create folder from the Create dropdown
Enter a folder name - I put a name 'grafana'. I suggest all lower case - unix case sensitivity etc.
Right click the folder, select properties and change owner to 'admin' if needed. also check the 'apply to sub-folders'. Save
Close File Station
Back on main screen select 'Docker'
On left menu select Registry.
on right side, enter 'grafana' in search and hit enter.
Hilight The grafana option at the top with the largest quantity of rating stars. This is the latest open source.
Select Download above it.
When prompted for Tag choose 'latest' - then 'Select'.
On the left menu, select Image. You should see the grafana/grafana - if the Disk icon next to its name is changing - it is still downloading. wait for it to complete. (the grafana/grafana:latest - I can't explain the redundancy but that is what you get)
Highlight grafana/grafana:latest and click 'Launch'
The first prompt is networks - select 'use the selected networks' and 'bridge' should be checked. then Next
Change the Container name to 'grafana'.
click 'Enable auto-restart'. Then Next. do not be tempted to click the web portal via web station!
Click the Advanced Settings button.
Under Environment, click Add
under variables column - in the blank put ' GF_AUTH_ANONYMOUS_ENABLED '. Under value put ' true ', and click Save.
Click Next.
On Port Settings screen enter 3000 under local port, then Next
Click 'Add Folder'. In the popup - drill to your folder name created above mine was 'SynologyNAS1 : docker : grafana' highlight it and click 'Select'.
Enter a mount path - use ' /var/lib/grafana ' and click 'Next'.
Review and uncheck 'Run the container after wizard finishes'. Click Done.
I'm on a windows machine - so I'll use a cmd window to run ssh. I open a cmd window and issue the following text commands:
ssh {NASIPAddr} -p22
The result will be a prompt for a password (look at the prompt for what user account it defaulted too) and enter the correct PW.
enter "sudo -i"
then "cd /volume1/docker" (or whatever volume/folder you're using) and enter
then "sudo chown 472 grafana" and enter
close your cmd window
Back on the NAS webpage, in Docker - go to 'Container' on the left.
Select the grafana on the right and change the toggle switch at the far right to 'on' (blue).

On the left menu - select Container - grafana should say 'running'.
choose Overview on the left, you should see grafana, and influxdb (and I also have Echo speaks from some other work!)
Close Docker

Access grafana on the NAS web screen usually http://{NASName}:3000 or http://xxx.xxx.xxx.xxx:3000
Since this is the first time, we have to put the default user/password, which is 'admin / admin'.
The next screen is to change the password - make note! - you should now see the Welcome screen.

To hook influxdb is next.
Go to your influxdb web page usually http://{NASName}:8086 or http://xxx.xxx.xxx.xxx:8086
On the left menu select the Load Data, and select 'API Tokens'.
On the right, select Generate API Token.
Select All Access API Token in the drop down - IMPORTANT. you only get to see this once! copy it somewhere!
For description I put 'TOKEN_FOR_GRAFANA' and click save.
COPY THIS STRING AND SAVE IT!

Access grafana on the NAS again usually http://{NASName}:3000 or http://xxx.xxx.xxx.xxx:3000
Select the Add Data Sources box near the middle of the screen.
Find 'InfluxDB' and select it.
The Data Sources/Influxdb opens: No need to change the name unless you want/need to.
For Query language, select Flux
For HTTP / URL put "http://xxx.xxx.xxx.xxx:8086" (Docker / DNS is easier to workaround by just using IP address)
Change all Auth options to OFF.
Under InfluxDB Details enter the you used with your Influxdb install organization. From the influx instructions before, I suggested city name, Capitlaized.
Get that copied token you made for grafana now and paste it in the Token field.
Now enter Hubitat or whatever you called the bucket from the Influxdb.
Click save and test.
If all has gone well - you get a green check mark and 'datasource is working. 3 buckets found' message.

Let's not stop! Let's get a dashboard showing data.
Click the top left Grafana icon, and return to the home page.
Select Dashboards - create your first ..'
Select Add a new panel
Flip back to your InfluxDB web page, and on the left select 'data explorer'
Select Hubitat (or whatever you called your bucket) and it should pop a 'filter' to the right, which shows your attributes you are logging.
Select one for now.
Click Submit on the right, middle of screen. You should see a basic graph.
Good so far? I hope. Next to Submit is 'Script Editor'. click that and you see a text version of your graph query. Copy this text.
Back now to grafana web page.
Paste the query string onto line 1 of your query and press enter.
Do you get a Graph? Save it by pushing the save file icon at the top right.

Lets test output on a tile - I use Hubitat Dashboard for Android which allows me to add a web tile and I just enter a URL.
Click the Share icon above your panel. I don't lock time range. Copy the Link URL into your dashboard tile.
et Viola!

2 Likes

Caveat: I've been using Influx v1.8 for several years and I'm having a hard time migrating my brain to v2.x.

My 1.8 instance runs natively on Arch Linux. The 2.6.1 instance I'm playing with is in Docker on the same hardware. I changed the config.yml line to "http-bind-address: :9086" to allow them in parallel.

I'm running version
image
of the logger on
image
with this config

It connects to my v2 instance just fine but the payload is not getting into the database due to a "204 No Content" error (token intentionally left out)
image

Here is the HE log of the transaction

Am I the bug here?

A response code of 204 from Influx DB is a success response. Where is that output coming from?

No errors are showing in that HE Log output. I suspect that the "No Content" message is that there is no body in the response.

So then the question is why is the data not showing up. If you go into the InfluxDB 2.x gui and pull the information from the "_monitoring" bucket to see what is shown.

I guess you're asking about the network traffic capture in Wireshark?

Neither of the battery or motion measurements are present in the list when I add a cell to a dashboard in the InfluxDB GUI
image

So it appears to me that the write didn't happen. :man_shrugging:

I'd like to query the database in a terminal to see what is actually there but I haven't figured out how to do that withe v2x yet. It's a piece of cake with v1.8.

Unfortunately, I don't think any of the data you provided points to a problem. It looks like for some reason InfluxDB 2.x isn't importing the data for some reason, but responding with a success to the HE. The No Content message is just that there is no response data being provided by influx other then the successful response code. I will do some testing with my instance since i am on Influxdb 2.6 myself and see what if i can figure out what is happening.

Is the database used with InfluxDB 2.x migrated data from the 1.x instance, or is it a completely new load?