Hub Load - Elevated/Severe

IIRC the engineering logs will persist though the shutdown/restart.

1 Like

To see if there was any correlation to when the load was normal/elevated/severe..
I created a pivot chart.. based on minute of hour.. and saw this..
image
that it was happening at minute number 1,6,11,16 ...
based on this info I looked at the scheduled tasks in logs.. and found that I had hub info device/driver polling the hub at those times..
So I deleted that virtual hub info device..
and then just to clean up.. shutdown the hub, removed the plug, waited 5 minutes and plugged it back in .. and then repaired/re-installed the hub info driver from HPM..
and..... amazingly now my hub is continuously running at less than 0.4 (5min CPU avg)
So I think my issue seems resolved.

In short, my guess is during the upgrade, the hub info device or driver got stuck/corrupted or something,
My solution was
-- to delete the hub info device..
-- then repairing the hub-info driver from HPM,
-- powering off and disconnecting the hub for 5 minutes
-- re adding the hub-info virtual device.

2 Likes

Interesting... @thebearmay - tagging in case you didn't see this and it is interesting/useful to you.

1 Like

Interesting. Did know that all of the drivers and apps were getting their definitions updated during the upgrades, and have seen instances where I needed to resave the preferences, but this is a new one.

1 Like

That shouldn't break anything, and I will get all the info I could get in the logs either way, so go ahead.

They all get loaded and their definitions are updated, if necessary, but they're not recompiled.
User apps/drivers are recompiled on demand only, when one hits Save button in the code editor.

That whole scenario is weird. It doesn't make sense to me, but it obviously is what it is. I'll keep it in mind in case another report like this pops up, maybe there's a pattern.

6 Likes

I guess..my resolution was short lived.. just about a 16 hours!
I was watching the hub throughout the day.. and :disappointed_relieved: hub elevated load message popped up..
So first thing I did was uninstalled the hub-info driver completely and rebooted the hub...
It went back to low numbers for 2-3 hours and back to high numbers

12-28 18:17:42,357976, 0.79
12-28 18:22:42,354316, 1.52
12-28 18:27:42,356204, 1.66
12-28 18:32:42,351268, 1.3
12-28 18:37:42,351480, 0.9
12-28 18:42:42,350544, 1.64
12-28 18:47:42,350884, 0.84
12-28 18:52:42,348512, 1.48
12-28 18:57:42,347628, 1.24
12-28 19:02:45,353176, 1.47
12-28 19:07:45,346828, 1.39
12-28 19:12:45,352564, 1.19
12-28 19:17:45,353268, 1.73
12-28 19:22:45,351316, 2.81
12-28 19:27:45,350252, 1.61
12-28 19:32:47,350856, 1.81
12-28 19:37:49,349692, 1.41
12-28 19:42:49,348828, 1.4
12-28 19:47:49,350692, 1.21
12-28 19:52:49,347776, 1.54
12-28 19:57:49,348152, 1.73
12-28 20:02:52,358248, 1.73

Although, I did update from .141 to .142.. dont know if there was any correlation to that (Maybe I shouldnt have)

So, I am back to square one.. @gopher.ny if you could take a look further, I would highly appreciate it !

Try a reboot and check the box to "rebuild database on reboot"

That should perform what is essentially a soft reset.

2 Likes

That's worth a shot.
I checked the hub a few times but yet to catch it in the "busy" state. Looks like it goes back and forth between running normally and spiking.

1 Like

Dong the reboot with db restore now..

1 Like

I see the spikes still happen on the hub.

I'm adding a few debug options for the next build. It will not fix anything but it will likely give me an idea of what the issue is.

2 Likes

Just now i see 2.805

If you are online.. maybe it can help.Also
I have setup a rule automation to notify me when i get a spike.. is there a way to add you to get notified to the automation so that you don't have to manually check for high spike!?

1 Like

great... thank you very much

1 Like

fyi..Updated my hub to .144

1 Like

any other ideas?
the slow automations, especially lighting has started getting on my nerves :frowning:

What I dont understand is that my apps and devices stats are below 3-5% consistently, but still the hub load is elevated (and severe at times, albeit fewer now than before )

is nuking and redoing everything my only option now?