Can someone help me understand how 4% and 1% is considered a severe load on my C7 hub?
Severe load is generally flagged when the CPU Load exceeds a normal value for multiple polling periods. CPU load is a basically a measure of running tasks + tasks waiting for response (generally IO or HTTP response). Hubs with apps or devices that do a lot of HTTP calls tend see this condition most often.
2 Likes
Is the best way to fix it to use the rebooter app to periodically reboot it?
That will mask the problem but not really solve it. May want to try grabbing the 5 minute load value from
http://yourHubIP/hub/advanced/freeOSMemoryLast
It will give you a couple of lines that look similar to:
Date/time,Free OS,5m CPU avg
2021-12-07 13:40:25,458788,0.01
(in this instance the average load over the last 5 minutes is 0.01)
then disable an app, wait for 5 minutes and re-poll the url to see if you had a drop in load. If not disable the next app and repeat until you see a noticeable difference. That should be your problem child.
3 Likes
I've finally gotten around to testing/working on this. I'm doing anything that requires wifi/cloud connection first.
I started by changing my Ecobee to only poll once per hour. That didn't help. I then disabled my Ambient Weather station in hubitat... still nothing.
I've moved my Chromecast integration to Node Red and it seemed to make the biggest change. However, it wasn't terribly significant.
Please see attached..I still don't understand how this eventually becomes a severe load. 5m average is always under .1, except for the first 10 mins after a reboot. During that first 10 mins it'll drop from 600,000 free mem... to about 490,000. It'll then slowly drop from 490,000.
I forget what the level is that triggers the severe load alert, but I do remember that it needs to be met or exceeded several times in a row (which is why you don't get it on a reboot), and I'm just not seeing that in the displays. Are you still getting that alert?
I am. I get it around every 5-7 days. I'm wondering if it was the Chromecast Integration. I just unloaded the Chromecast integration yesterday so I'm waiting to see if/when the severe load triggers.
I thought it was my 5 minute polls of either my ecobee or weather station... The Harmony hub is the only other thing i have on a cloud and i haven't gotten to testing that yet, but it rarely makes calls to Logitech and is never in the log, app or device stats...
I do have several Maker APIs setup for different node red servers and a couple of apps on the phone but they don't ever touch my app/device stats.
The top three consistently is the Ecobee with 7 sensors, my PWS and the Google Homes. There's 8 Google Homes and they each found their way into the top of device stats. The Ecobee and its sensors do the same but i didn't see much of a change when i changed polling to once an hour. So the 8 combined Homes far exceed the load of the Ecobee and the PWS. When I removed them device stats went from .8% to .1%. App Stats went from 2.4% to 1.6%... so quite significant there... but not much of a change in the 5m average.
Any ideas on this one @gopher.ny ?
Oh yeah I've also unloaded my GCal integration and moved it into IFTTT, which had zero effect.
Can you PM me your hub's id? I'll take a look.
1 Like
Done... thank you!
Nothing jumps out in the logs until errors start popping up, unfortunately.
I wonder what those apps/devices tabs in Logs looks like while it's alerting. Judging by how low the usual load is, we may have a clear "winner" after just a few minutes of "high load" state.
2 Likes
I'm watching everything religiously currently to see what could be triggering it. I'll let you know when I see it again.
2 Likes
Probably not much help here, but the only time I've seen app execution times any where near your 'Volume to 11' time was when I had a malfunctioning Google Mini. Using Wireguard I found the thing was phoning home every two seconds, which was enough to clog the LAN, delaying other external calls, and eventually slowing the hub to a crawl.
1 Like
That's interesting. All that is, is a RM rule that is triggered when a button is pressed. It tells my Harmony Hub to send 40 volume down commands to my tv and then 11 up. It's a work around to automate a set volume on my receiver.
I just ran my RM "volume to 18" with similar stats...its a slow process to input 51-58 commands so that seems fair. These rules don't run consistently over a period of time so I'm not sure if it would trigger the severe load. Load during that 5m was only .06.
Your reply makes me wonder if it was a Chromecast integration issue. I'll know soon enough with Node Red controlling them currently.
1 Like
@bjcowles This would also make sense with the amount of Google Homes I have. It could explain other symptoms I've noticed, as well. My speedtest will run at 1gig but things don't seem to load at 1gig. Last week my mesh network went kind of AWOL. Xfinity took blame and "sent a signal." After several resets it appears to run normally now. Could be the LAN getting clogged?
My issue is I have Wireshark but it's extremely foreign to me. I have 45 devices on my LAN as well and use A LOT of data...4 tvs streaming, tablets, phones, laptops, security cameras (nest), work computers/phones on their own vpn that make A LOT of calls home. So my network is producing A LOT of data. If I have an issue on the LAN it's going to be like finding a needle in haystack.
I'm guessing I'll get a severe load alert soon. See attached. It goes from an average load of below .1 but is now consistently above 1? I'm also not seeing much change in the log, device stats or app stats.
I'm curious if I found the issue. When I turned on my good morning switch RM sent several double commands within .005 seconds to the Ecobee. Ecobee doesn't like getting overloaded with quick commands and will act up. I put debugging at level 5 for the Ecobee manager to see if it's giving anything I'm not seeing at the normal logging level.
The Ecobee Manager seems to be logging out periodically from my Ecobee account. It's running a watchdog even while logged out like every 1-2 seconds. I'm not sure if this normal as I don't want to reboot and see if this is something still happening, I still don't have the load alert. I don't want to remove the Ecobee integration as that will break A LOT of rules.
I just got a notification for excessive events generated by the Ecobee Thermostat in the last hour. I think my problem child is surfacing.
I experienced the same issues back last year. Normal activity for a few days, then steady increase in processor until severe load warning.
The problem was, at least to me, ecobee suite. I now use the standard integration with no problems, I just had to get creative with rule machine for some stuff. You can PM me if you want to chat,
2 Likes
