I've just compared about 100 lines of logging between the on-hub event log, the Maker API URL external log, and the websocket (as dumped by the alpha version of helog) method, and both the on-hub event log and the Maker API logging method are missing events. It seems to be when a bunch of events happen "at the same time" (on the same second), some of them get lost, both in the list of events recorded on the hub and in the events sent out via Maker API (which makes sense, since Maker API reads the on-hub event stream). It doesn't explain why the websocket would have more events than the on-hub log.
Since the on-hub event stream is missing events, that may explain why some automation isn't working the way I want it to. The events I'm logging (and losing) are from an Aeotec Energy Meter Gen5 and it produces a lot of events. In my small sample, I also see lost events from the Hub Info driver.
Lucky for me, I have all these past log records to compare with.
In any case, it means I have to give up on using Maker API URL logging, and give up on the idea of downloading events from the hub, since they are both incomplete. Only the (UNSUPPORTED) websocket appears to have all the events.
I'll have to upgrade my Java skills (and learn the Gradle build system?) so I can modify the 495-line Java program helog to add the missing date information and append log lines to my event log file. Or maybe I can just find a C program that does the same thing in 50 lines.
Aye. I wish events had time stamps. Neither the events sent through Maker API nor the events sent through eventsocket have time stamps. I have not enabled logging for most of my devices and apps. I prefer to leave the logs for errors and warnings.
I ran a test and created 100 concurrent connections to xinetd and it handled them all without error or data loss, so the missing events are likely the fault of Hubitat not sending them, not my logging host being to slow or too resource-limited to log them. Given that the events don't even appear in the on-hub event listing for the device, I'd say Hubitat isn't sending the events correctly. The events appear in the eventsocket, so I know the driver is generating them; they just don't make it into the on-hub event listing (and therefore are never sent through Maker API).
Is there some setting I'm supposed to turn on to have all the events recorded in the on-hub event list?
Are any of the missing events duplicates? Hubitat’s platform typically deduplicates events, unless the driver explicitly forces an event for every status update. Button controller devices are an example of a device that always generates an event for each button press, even if the event is the exact same as the previous one. Whereas a light switch claiming it was turned on multiple times in a row, without the corresponding off events, would result in only one “on” event being generated. This is due to the fact that some devices report their status routinely, even if nothing has changed. As events are used to trigger apps, it does not usually make sense to generate events for duplicate status updates.
Aha, yes, the events that are missing are duplicates of previously logged events. I missed that. If any of the event values being reported match a previously reported event value, no matter how long ago it was reported, the event is suppressed. I guess duplicate event suppression doesn't apply to the eventsocket stream?
While suppressing duplicates makes sense for home automation events, as you say, it does make graphing of sensor report events using the event logs awkward, since you may have to look back to the "beginning of time" in the event logs to know the current value of a sensor, if the value hasn't changed in ages.
I found a nice little Perl library that lets me dump a websocket. I don't have to learn Java after all.
101 lines in one file (instead of 495 lines in nine files). Less code is better code. Thanks to @aaiyar for suggesting using the unsupported and undocumented websocket interface. Feature Request: Make this a supported feature and document it.
I note that the Maker API events lag the eventsocket events by about 10 seconds.
P.S. Another awkwardness is that the eventsocket JSON format differs from the Maker API JSON format. The Maker API JSON format has a useless outer {"content":{ wrapper that the eventsocket JSON doesn't. The eventsocket JSON doesn't put double quotes around single digit numbers, but Maker API JSON does. One has source, the other has data. Of course a bit of Perl code can make them the same so they can go into the same log file, but why do they need to be different?
I'm seeing strange character set issues with the eventsocket records. For temperature events sent via the Maker API URL, all the "degree" (°) characters are in proper UTF-8 format (\xC2\xB0) in all the fields in all the records, e.g.
... "descriptionText":"Tuya-1 temperature is 16.69°C" ... "unit":"°C"
Those same records coming from eventsocket have the first degree character in the unit field in Latin-1 (\xB0) and the second one in the descriptionText field appears as UTF-16 escaped using a six-character ASCII sequence starting with a backslash: \u00B0:
... "descriptionText" : "Tuya-1 temperature is 17.00\u00B0C"
This UTF-16 escape also appears when the degree symbol occurs in the value field in other records, e.g. from Hub Info:
... "value" : "96.8 \u00B0F"
I'll write Perl code to fix both the Latin-1 and the ASCII escape to UTF-8 as I read records from the eventsocket, but why is this fix needed? Why isn't the character set identical to the records saved by Maker API?
Even 33,000 events isn't a big enough number. This Energy Meter device has 18 states that it reports on (six sets of three), so 33,000 entries is only about 1,800 entries per event type (and Hubitat sets a limit of 2,000 entries for the same event type).
For an event that reports every 10 seconds, a full 2,000 entries (the maximum allowed) is only 5.5 hours. That's not enough back-up; I want at least 24 hours.
I reconfigured my network yesterday and the process that reads the websocket hung and stopped recording events. I didn't notice until 18 hours later. I used that huge URL nonsense to grab the entire event log for each of the devices (Feature Request:A button to download all the events), but for the events that repeat every 10 seconds I lost 12.5 hours of entries because I was only permitted 2,000 (5.5 hours) of them to be saved.
Alas, Hubitat doesn't let me do what I want, which is to save 24 hours of each event, nor does it let me save different numbers of events depending on the frequency of reporting. For events that repeat hourly, 2,000 entries is an unnecessary 2,000 hours (83 days). For sensor logs that repeat every 10 seconds, 2,000 is not nearly enough.
So what is the problem you are trying to address or your use case.
Why does Hubitat need to handle this?
Have you tried adjusting the event retention on your devices. Like set them to a few thousand instead of the smaller number they have normally. Ofcourse that may have a negative impact on the hub, but there really isnt a way for this ask not to if you want it all to stay on the local device.
Keep in mind Hubitat isn't a analytics machine but a event based Home Automation system. Frankly once a event is processed from a home Automation perspective the event info is useless.
I want to collect sensor data from this Energy Meter. I also want Hubitat to know some of these values for use in Dashboard tiles. I want the device to which the Energy Meter is connected (Hubitat) to store 24 hours worth of logs in case of problems with the external server that is also collecting the sensor data.
Because as far as I know I can't connect a ZWave device to two different hubs at the same time, so the device to which it connects is the one responsible for keeping the back-up event trail. I could connect the Energy Meter to some other device, but then I'd need to have a way to communicate the current values of sensors to Hubitat so that Hubitat could act on them.
If Hubitat proves incapable of doing the job, I can look into open source alternatives where I can program the retention period, but that's work. For the moment Hubitat almost works, and I have not abandoned all hopes for its continued improvement.
As I said, I have the limit set to 2,000 for all devices and Hubitat prevents me from going higher to satisfy my need for 24 hours of retained data. As I explained, 2,000 entries is more than is needed for some sensor data, and is too short by half for other sensor data.
What does this have to do with Home Automation? It keeps a trail, but the data is useless after the event happens and the hub acts on it.
Have you considered using a solution pointed out above that is designed to keep analytic data for events over a long period of time.
As stated earlier I use InfluxDB to keep event data. I use a Smart app called InfluxDB Logger to put all event data into InfluxDB. Then I use Grafana to be able to create dashboards and show data about what the events reported.
I also have a flow in Node red that reads the live Logging websocket and puts it all in a syslog format for review later in InfluxDB as well. I have a dashboard in Grafana that allows me to review that syslog data.
The key with this is using the right tool for the job. InfluxDB is a database designed completely around be a time based event database. Grafana is a analytics engine for visualizing pretty much anything.
The reason this data isn't kept in the hub DB is that it would eventually slow the hub down enough were people would see bad performance impacts.
It's not useless to me; I needed that trail just yesterday and having it allowed me to restore missed log entries for many of my sensor events.
The Energy Meter device can only be connected to one hub, so that's where I want the back-up copy of the sensor log to be. If Hubitat can't keep the backup sensor log, I can get another kind of hub that can do the job, but then I have to interconnect the two hubs to make automation work and that's more work than I want to do today.
I've already said that I use a websocket to pull data off the hub. That works fine, except when it doesn't and I need to use the trail to recover missing sensor log entries. If sending off-hub were a flawless process, if I never rebooted the machine collecting the long-term logs, if the network never had interruptions, I wouldn't need a trail. But I do need it.
Send me some figures on how keeping a full list of events slows down the hub; I don't see it. I would be happy to see my hub slow down a little because it's being used. At the moment it's idle most of the time, even though I'm storing 2,000 entries for all my events on all my devices.
The whole point of a database is to make the storage and retrieval of information efficient even at scale. Hubitat is doing a fine job, except that I need more.
Have you considered using an IoTaWatt energy monitoring device? It can export data to InfluxDB, PVoutput, and Emoncms using a local onboard store and forward buffer to essentially guarantee no data loss. I have two of them and they are extremely reliable. I also wrote and integration for Hubitat to allow the data to be used for home automation tasks (not historical logging.)
Perhaps I should have gone this route. I had a cursory impression that IoTaWatt was more geared to monitoring individual circuits. That would be great, but for me, I don't see how I could squeeze all those CTs in my breaker box, even though they may be small. The large CTs that came with the Aeotec, good for 200 amps, seem quite nice and are stacked up on the ones from my ancient Energy Detective, which has been operating great over the electric wires for quite a few years now.
I wouldn't mind maybe doing some analysis, but that'll take a lot of studying for a dumb**s like me.
All this tells me is that you want that data on the hub and not why. It seems to me that you just don't want to consider using a external system. Yes anytime you have to send the transactions somewhere there is a chance that records could be lost while doing maintenace, but I have yet to loose records simply out of the blue. it is always because I am doing work on my server and made it unavailable for one reason or another. if it is so critical that you can't have that happen then invest in redundancy so the external DB is always available. You can also use a cloud based InfluxDB database so that it is your provider make sure it is available.
Hubitat has one database. It runs on flash and the system is ARM based. It doesn't have limitless cpu capacity or storage. An 11 year old Core i5 with 8GB of ram and a 512GB SSD will blow the Hubitat out of the water for this data capture and analysis use case. That said i would ask is this really the right use of the hubs resources.
And that is just one way to get the data off the hub. A smartapp can also do a great job of it. All of the device statistics I collect are a great example of that.
That is something that the Hubitat folks would need to give you. But there are a bunch of threads that reference lowering event retention to improve performance and though have a response go from .1 seconds to .5 may not bother you, there are allot of folks that don't want a half second or 1 second delay for their lights to turn on when they go into a room if it can be prevented by simply running a clean database.
This is kind of true but not for all use cases with the same database. You have databases that are OLTP (Online Transactional Processing) and OLAP (Online Analytical Processing) that are designed and optimized for different kinds of work loads. IBM has a article that discusses the difference here.
Either way I feel like I have given you a valid workable solution based on the information you have given so good luck.
I think this thread received plenty of input for the initial question. Thanks everyone for your feedback. If you have further questions please create a new thread, as this one has gone astray and will now be closed.
