Suddenly every device generates excessive hub load

While the rebuild of the database is an great idea, and definitely the proper starting point. I'm not a huge fan of just "nightly or weekly reboots" - That said, I AM a fan of "rule triggered" reboot (automagically), when the hub is headed towards issues - aka reboot before things get bad, and 63 days is quite a bit of uptime, given that hub still has some memory leaks.

For me, I "schedule" reboots, when free memory gets under 180K or so (I use this limit on both a C7 and C8 - No C8P involved). Given my app/driver mix, that fairly routinely works out to about 21-28 days of uptime, again, this is very much driven by what drivers and applications your using, and how often they are active. My hub would never make it past 6 weeks, and the failure scenarios would be basically what you described.

So my approach, is that I monitor the free memory as part of the HubInfo V3 driver/device (I think I have it polling every 10 mins or so). I think the hub can actually still run in the low 100K memory range, and after a reboot it's around 400K, after running a few hours, it works it's way into the high 300K range, then it's a slow/gradual, but continual drop. And for me, under 100K bad things start to happen.

So I have a rule, that sets a "reboot needed" hub variable boolean, when the free mem drops below 180 K - The reboot doesn't occur, at that time, as that may be the middle of the day, or I'm using the hub, etc. - But I have a rule that runs at 3:35 AM (avoiding a bunch of built in hub cleanup that happens around 1 or 2 AM), if the "reboot needed" flag is set, this rule triggers the "reboot command" (also supplied by the HubInfo driver). And if the hub reboots, for any reason, the "reboot needed" flag is cleared, and a "reboot occured" flag is set. - Then at 8AM, an alert is sent to my phone, if anything was rebooted during the night.

To be honest, given upgrades and other changes, I often manually reboot before the 24 day mark, so I only get this alert 5-6 times a year.

Bottom line, setting the hub up to reboot automatically when it's in "problem states" (low memort, cloud backup failed, radio offline, etc.) - And you can decide what states "merit" a reboot. - Then a reboot scheduled during a "quiet time" later that day, and you're set for another few weeks. - Personally, I find daily or even weekly reboots indictative of some more serious problems. But you NOT going to get months/years of uptime with a C7/C8 (Perhaps with the extra RAM of a C8P, you could go for months) but the hub does have some memory leaks given it's basically Java based, doing GC, and can run user code (which isn't always perfect around releasing resources). So 63 days of uptime, is alot, based on my experiences.

Hope that helps with my view from the trenches..

3 Likes

Thanks, great explanation. I like the idea of doing reboots based on data instead of a rigid schedule.

I've now installed the HubInfo driver and will also add the related Hub Monitor program. I'll work on a similar set of rules to reboot my hub.

Please try this to see if it solves your problems.