Reliability issues and firmware updates

I haven't changed anything with my Hubitat in weeks except for running the latest firmware updates and I'm having issues in the last week with inconsistent performance of rules. I have motion detection and lighting rules that are only working some of the time. Has anyone else experienced reliability issues in the last week or two?

Absolutely I have. Firmware 2.1.7 and beyond is garbage for performance among other problems.

2 Likes

Did you end up rolling back to a previous version?

I was rolling back to 2.1.6 each time I tried the newest and the problems remained. But unfortunately you can only rollback so far back. I can no longer rollback to 2.1.6

Something to keep in mind is that the system might need some time to settle after an update was performed. Without first hand knowledge, I can only assume that there are some background tasks that need to be performed with an update and that this might also be dependent on the version.

Here is a graph of my central hub (no devices attached, all kinds of custom apps, ton's of LAN based integrations, good amount of rules)

My average response time to loading the "Apps" page prior to the update was between 500 - 600 ms. I measure that on a consistent basis.
I performed the version update at around 7:45 PM, and after that was done, I did see a spike in the average load time. The spike was big enough to notice when working with the system. I did not do anything to fix this (outside of going to bed and sleep over it :slight_smile: ) but as you can see, my average load times went down again and the hub is as fast as always.

Just as a note, this my C4 hub. I do see a similar behavior on my C5 hub but the load times are in general much higher. I have two C4s and two C5s and there is a night and day difference between the average load time.

All of this in without proof and pure assumption based on observation.

3 Likes

I agree with your assessment. I've found that a second reboot after the upgrade is complete helps with this. However these reported latency issues are a separate matter, they continue beyond weeks after upgrade. And frankly, I've been dealing with them since purchase over a year ago with a lot of things taking blame that didn't pan out to be the problem. The one thing that doesn't seem to take blame is the firmware itself. I don't accept that anymore, I've proved otherwise. I'd be cracking skulls if my software team kept telling me it works on my machine when customers continue to report performance issues. It's time to stop enhancing and start digging into why there is a problem among the community that has been consistently reporting it for a year now.

I'm saddened to hear the C5 performs so much better than the C4, mostly because there is no way I'm purchasing another hub until this is resolved on my two existing.

5 Likes

I am actually saying the other way around is better. :slight_smile:

I didn't mean to take anything away from your experience. I do believe that there are slowdowns due to something. I just wanted to provide one train of thought to not always judge the performance of one update right after the update was installed.

I am 100% with you on that one, I would never accept that scenario myself with my team. Sometimes stability/performance can be seen as a "feature", if you simply look at it from a rollout perspective. IMHO, it is an important feature.

I am sure that there is more to it than we might think or know. In my opinion, I don't think the Hubitat team denies that there is something... They just can put their finger on it or would be solved by now.

Oh, then yay!

Which is why it's time to buckle down and do it, or find someone that can. I'm out of patience and about to discontinue use.

6 Likes

How often do you see these slowdowns / issues. While not perfect, do you think a night reboot might keep you from jumping off the edge?

EDIT: Just as a reference, I did recently install a NUT UPS driver that was causing my hubs to come to unfortunate Server 500 error. After some investigating, I figured out that for each refresh that driver was establishing a new telnet session without closing the original one. That was done every 30 seconds.... I modified the driver to keep the connection established instead creating a new one all the time and voila, no more issues. Where am I going with this: When your issues occur, do you see anything in the logs? Like "Can't access database" issues or something like that? And no, I am not going the "it is always a custom app/drivers fault". It is just an easy way for me to visualize how I started to track down my issue

I've noticed the same thing in the last few weeks and I haven't updated firmware in a long time. I'm at 2.1.5 and the only thing I've done was add another Konnected board so I tried disabling the entire Konnected app and rebooting which didn't help I then backup my hub then restored from the backup to try to straighten out any DB issues and rebooted. That seems to have helped some at least, still not sure its performing as well as it was 3 weeks ago. I didn't see errors in the logs at all.

Ditto. :cry:

Intermittently through out every day and night, the latency between motion active received at the hub and the fired event can be up to 30 seconds, reboots do not prevent it happening.

It's definitively not a custom driver or app problem. I've done due diligence. I stripped the hub down to native. The same problem occurs in RM, Simple Lighting and Motion Lighting with only native drivers and no custom apps or drivers installed.

When that didn't work, I then wrote my own apps so I could catch and log the events, that's how I know the hub is receiving the message from the device and is delayed in taking action, delayed firing the events based on the device subscriptions. I'm also seeing problems with state, lights will not physically turn on but the attribute of the light shows on. And vice versa, the light will physically turn off, but the attribute remains on.

It's not mesh related. It's not device specific, the HE team touts Lutron as the most reliable, so I installed Lutron in problem areas within feet of the Lutron bridge, same problem.

We've had great developers quit the community, because their apps were crapped on as source of the problem, yet the problem persists without. It's time I see this become the number one priority or it's time for me to move on. I hope it's the former, as I've invested a lot in this community and would hate to see this fail.

7 Likes

WOW! I thought mine with 3-4 seconds was bad. 30 seconds?!? That's nuts. Do you have one hub or several?

I have two.

It's not always that extreme, but it can be. And it stacks. Passing through a room to the next, 15 seconds for one, 15 seconds for the next, so a total of 30 for the final room. IOW, pass through the master bedroom to the master bath, the master bedroom will come on 15 seconds after I leave the room. I'm in the master bathroom with in a second or so of having entered and left the bedroom and the master bath will wait the 15 seconds for the bedroom and another 15 seconds of it's own.

That's an extreme example, but if you're moving from one side of the house to the other, it's very pronounced. Worse yet, the more activity in the house, the worse it becomes. The hub becomes bogged at times. Keeping in mind, however, at other points during the day, the same level of activity or passthrough will not produce any latency at all.

How are your devices/apps divided? Have you considered experimenting with going back to a single hub to see if there's a change? I'm a member of the "One Hub Is Enough" club so I'm dying to see if someone is able to finally figure out if there is an appreciable benefit from using multiple hubs or if it really is a detriment, as I've suspected.

I would totally expect that. The more activity, the more events. The more apps that have subscriptions to those events, the longer it's going to take to process them.

It's the same with Hub Connect. Even if you don't have an app running for events...if the device is synced with hub connect, it has to spin up for every single event the hub generates. Apps only spin up for those that they need and subscribe to.

I've sliced it a couple of ways. Initially the plan was the split the devices between areas. I did that, to no effect. Then I offloaded all custom apps and drivers to one hub and went native on the other, to no effect. Right now, I have a certain subset of activities on one hub, all of lighting and motion on the other. Both hubs, no matter what configuration I had them in, suffered similar latency and lock ups. To that end, I believe you to be right, one hub should be enough, two didn't improve it.

I do have HubConnect, because I must. Otherwise, I'm very conservative about this. I don't tolerate apps on the lighting hub (the most critical for performance) that are written poorly. Event Subscriptions need to be handled with care, scheduled events with even more care. Since I'm a developer, I'm able to be very purposeful with this.

HubConnect certainly cannot be the problem however, because I went to two hubs only when I couldn't solve the problem with one.

Are both, Zigbee and Zwave, behaving that way?

One day you will give in :wink:

2 Likes

I'm using a single hub as well.

I use Device Watchdog but I added almost everything as a battery device so that it would monitor those updates only.

I do use Hub Connect, never thought about that having to report at the same time as I haven't had performance issues until a few weeks ago with no real changes, although I did add a few new devices in Hub Connect when I added the second Konnected board.