Source of "Severe hub CPU load detected"

I have to agree with everyone, your stats seem high. My uptime changes whenever the nightly Hubitat Database Cleanup/Backup process runs so as it ran early this morning my uptime only shows as 9 hours, BUT my stats are as follows -

Uptime: 9h 9m 28s
Devices: 0h 5m 14s (-0.0%)
Local apps: 0h 15m 49s (-0.0%)

This is on a C4 running 90 devices. I used to have to reboot frequently (once a day or every couple of days) as my hub would either slow down to the point of being useless or freeze completely. One of the first things I did was to remove my Ecobee from Hubitat and that helped a lot, but did not fix the problem completely, still had to reboot once or twice a week. I then did what @erktrek mentioned and offloaded all of my rules to an external server (node-red). That stabilized my hub significantly. Next step was to go through and determine if I really needed every one of the apps I had running. These steps over a period of 3-4 months got me to where I am now. The only time I reboot now, is if I decide that an update to the HE code is worth me installing it.

2 Likes

I do, but even if you add my 2 hubs together the worst case loading is about 20x less than yours.

I have ~150 devices across the 2 hubs.

i have higher stats about 5 % for apps and 4.5 for devices. .the apps stats are misleading and eroneous for web connections like my email , etc. because it logs the entire time it takes to connect and send an email but that is mostly cloud delays not really hub usage.

3 Likes

I used to reboot daily to avoid hub slowdowns prior to adding a second HE hub. It has been over a year now since adding the second one and I haven’t regretted it. I do try to avoid WiFi devices like the plague though since they seem to eat up the hub’s cpu time. For instance, my original hub is running Echo Speaks with 10 Alexa devices plus around 40 Zigbee and 23 z-wave showing ~1.3% app and device usage with ES (app and devices) using around 1/3 of that which doesn’t seem unreasonable. One my second hub I have 6 TP-link devices and 70+ Zigbee, plus most of my RM and ML rules, as well as CoCoHue with 2 Hue hubs and over 300 total devices. Somehow the TP-link devices are using 1/3 of the cpu time on the hub with no energy reporting which seems ridiculous. My hub load is still only a little over 3% though. I haven’t had to reboot since 2.2.3 was released.
Edit: I also reduced the state and event history sizes to 5 on most devices

2 Likes

So after asking on the Ecobee thread, it looks like that app's CPU load is somewhat normal. I haven't heard back about the WeMo app, but with Wemo being what it is, I would expect it to be normal as well.
I guess this, plus WebCore, which i use to run everything, is the cause of my issue.

I do wish there was a way to dig a bit more into this. Real time data in combination with historical data would be very helpful in pinpointing which event is causing an app to misbehave.

some of my tplink/kasa devices are high too, but again i believe that is time waiting for them to respond and some are slow at responding and have longer ping times, I dont think that really impacts cpu as the cpu should not be busy waiting during that time. That is why I said it is not 100% accurate expecially for cloud devics. I believe the stats are simplistic and count the elapsed time from startup to shutdown.. I have seen that with email where it may take 1-2 seconds for the port to open wheres during that time the cpu is not really busy.

1 Like

Yes, I ended up going with a lot of wifi devices and while there are certain advantages to it, there is definitely a higher overhead price to pay.
I'll try reducing the state and event history,

1 Like

any cloud implementation that takes a long time to run due to cloud delays is always going to show a high usage with the current implementation depending on how long the app /device is sitting there waiting for the cloud or internet to respond.

1 Like

That makes sense, but at the same time i think something must be spiking the CPU as reflected by the performance degradation.
Although it is hard to speculate since I have no clue what threshold constitutes a severe CPU load.

Well... You have the historical... You can click "Reset" to get (essentially) pseudo real-time.

2 Likes

Kind of, but if the CPU spikes for 5 sec and idles for 5, I'd get a reading of 50% which would be rather misleading.
I guess the systems events tab lists what occurred before the warning, but I haven't been able to reproduce the issue by re-trigering the events that occurred before the CPU warning.

You can also check the memory stats with this...

http://[your hub ip]/hub/advanced/freeOSMemoryHistory

Paste it into a spreadsheet of your choice and generate something like this:

Am watching my C-7 free memory trend downward... but the JVM stuff looks stable.

Better yet import into something like Grafana and you can track lots of different stuff. Search this community for more info..

BTW, all of my Kasa devices are local (disconnected from the TP-Link cloud). I wonder if part of it is just a consequence of using Wi-Fi .

mine are not using the cloud either.. but still are high.. i just think they are slow to respond.. i recently replaced a couple of them that are plugged in behind an ups and seem over time to have suffered from signal degradation.. The new one in the same spot is working much better. I also noticed that the latest version of the tplink/kasa integration was substantially slower after the latest round of changes. Not sure why . I went back to the older version that had no option for cloud integration.

1 Like

This seemed to make a big difference for my hubs. If nothing else, it dramatically reduced the size of backups and how much time it would take to accomplish. I can now hit Backup and my file is ready immediately. This hub has 323 devices:

If you ever wanted to add more complication and fun to your life you could run Home Assistant and integrate the TPLink devices there and then bridge them to HE...

Edit: I would likely not do this - instead probably just integrate in with Node-RED and control them there but that's for my setup. It is an option though and others have done this with various other integrations.

I noticed that as well. I subsequently decided it was time to start weeding out these devices and have replaced two lights (and some Sylvania A19s) with more Hue RGBWs that were on clearance at the local Costco, but the outlets are what really seem to be the issue.

Trust me that it is much more fun replacing them with Hue lights and zigbee outlets.

1 Like

I haven't gotten in to HA yet - I do have a server running on an RPi3 for experimentation. This looks interesting as a bridge..

1 Like

That was exactly why I wanted this integration. I have 36 kasa switch and integrating them into hubitat with a faster polling rate would kill my hub within 2 hours . By installing them in HA, I free HE from the heavy polling and have them report only when there is a state change.

HA poll the switch every 30 sec. so they are somewhat usable as a trigger in RM.

3 Likes