Elevated and Severe Hub Loads with v2.3.4

So am i and i wont say i’ve turned my back on HomeBridge completely, but so far Hubitat has provided me with some HomeKit functionality that no HomeBridge was able to provide so far without adding hardware to the RPI, namely Zwave and Zigbee device exposed to Homekit.

Im on beta 2.3.4.142 now and since 139 ive had no issues whatsoever (besides some rendering issues of overlapping text)

2 Likes

I can see why people are pissed off, normally hubitat is very closed off and secretive about their internal workings. Even large vendors provide a one or two liner when something goes haywire, not just "it's fixed". That and the fact that your own hub configuration data is encrypted, is a huge turn off. Then again this isnt an open source product, but the company hubitat does enjoy the benefits of the open source community, i.e. countless end users providing their time and labor writing custom integrations and drivers and sharing that publicly.

Personally I think it's B.S., but as long as the hub doesn't give me too much trouble the incentive to switch away from hubitat is minimal. That being said, it's not exactly a healthy or positive relationship that hubitat as an org is engendering with it's users.

If a competitor were to come along that addresses these issues, I'm guessing some people wouldn't have too much trouble making the switch.

So why don't you try being a bit more open as an organization and not quite as dismissive hubitat inc.

2 Likes

Count me in the group of users that doesn’t need to know how the sausage gets made.

Kudos to @gopher.ny and the rest of the team for identifying and correcting whatever details under the hood led to some (but not all) users to encounter this issue.

7 Likes

I get the frustration of those affected by the issue.

We (engineers) do not have direct access to the hubs. That is by design. We can get to the engineering logs, which are sometimes helpful and sometimes are not. Often, as it was in this case, we can't reproduce the issue internally. So we make a guess about what the issue is, add some logging in the next build, and make changes to address the suspected root cause. Sometimes we get it on the first try, but not this time. We're not happy about you having those issues, and we do our best to address them, within the constraints the lack of access imposes.

If you're really curious, we narrowed down the source of severe loads to use of Java's ProcessBuilder's redirectOutput and redirectErrorStream methods. It breaks down on a few hubs but not others, even though software is identical. We don't know why it happens, but we know not to use these going forward.

22 Likes

This is something that I wholeheartedly believe. I know that you have been very responsive to issues that I have encountered in the past and cannot thank you enough for what I can only describe as the most personal customer service that I have ever encountered.
Edit: Also, for an additional data point to the topic of this thread, I have seen a noticeable improvement in hub performance/increased free memory/decreased cpu usage even when compared to 2.3.3. I had been having intermittent problems with free memory dropping/ cpu usage increasing since 2.3.2, but couldn’t see anything that was reproducible. This appears to be fixed.

3 Likes

First class response right here. :+1:

Thank you kindly, sir! Really do appreciate it. :grinning:

6 Likes

I’m also using Homebridge and the Homekit beta on both hubs (and Hub Mesh), plus Echo Speaks and the Alexa integration on the primary hub and 2 instances of CoCoHue on the other. The current firmware has been the most stable one for me since 2.3.1. Of course, YMMV. I hope that you have a similar experience with it.

4 Likes

Thanks for the response and the details. I think that is a perfectly acceptable explanation and appreciate your time.

Also, as a sysadmin, I really get the frustration of an error that's hard to reproduce especially when it's only affecting some of what should be otherwise identical situations/configurations.

8 Likes

I suffered through the Vera “Xmas tree lights of death” issue for multiple weeks, that was painful!

This issue could at least be mitigated with an RM rule.

That said, I built and run the Incident, Problem and Service Level function for Flexera. Most of our software is closed source and we still publish a Post-mortem to tell our customers in general terms why a service broke and what we are doing to prevent it from happening again.

@mike.maxwell @gopher.ny @bobbyD it would provide peace of mind to customers if Hubitat could be more transparent about serious issues that impact customers. Thankfully these are very rare, but no team gets everything right, every time.

1 Like
3 Likes

Still seem to have a memory leak with the latest .139

1 Like

My hub has been much, much better since .139

Free memory significantly better and (so far) no recurrence of Z-wave failures.

5 Likes

Just got a memory low warning. Down to 80mb of free memory. Again, seem to have a pretty bad memory leak. Hopefully this will be addressed in future revisions.

For now have to reboot about once a week.

1 Like

Have you looked at the apps you have running to see if there is something in them that could be causing memory consumption. Memory isn't used for no reason, but does increase over time as more demand is found to use it. Many folks here are able to go the whole time between firmware updates without needing to reboot weekly. You may want to start a new thread about your concern for a memory leak as this issue with CPU consumption seems to have been addressed.

Appreciate the feedback. Nothing has changed on my end throughout the whole firmware issues with high memory usage. Suspect it’s still a firmware issue.

1 Like

I don't know the size of your network or number of automations you have, but I was having the same issues and have over 60 zigbee devices and 30+ zwave devices. I had a large number of automations and utilized homebridge and google home community. I ended up finally biting the bullet and rebuild from the ground up. It was certainly an extreme solution, and it did NOT result in correction of the CPU overload issues, UNTIL .139 came out. So, .139 fixed mine, but it may also have been a result of the rebuild. Many of my automation have been running for over 4 years. I may have very well had a corruption somewhere.
In the end, while disappointed I had to do this, I updated to the newer rule machine automations and integrated the Room Lighting app with success. I am ultimately happy with the end result. Kinda like a "Spring Cleaning" of my home automation network.

Perhaps a Spring Cleaning would benefit you, though, it's certainly not something that is fun or convenient...

I appreciate the suggestions and feedback. To do a complete redo is not something I am interested in doing at this point. It’s a major hassle. I will wait and see what future firmware revisions bring

Why don’t you just setup Rule to reboot when memory is low. Makes it self healing. I use 205MB, but you could adjust that easily based on your hub performance.

2 Likes

Thank you for the suggestion

1 Like

I’m happy to provide a copy of my code you if would like it?

All computers need regular reboots at some point, even my Linux servers get a monthly scheduled reboot.

EDIT: Here it is:

3 Likes