Elevated and Severe Hub Loads with v2.3.4

🛎️ Get Help
7

There's CoCoHue and a couple of my own drivers which interpret MQTT feeds, but that's all.

Here are some device stats:

image
image1580×1343 128 KB

And here are all of the app stats:

image
image1603×954 180 KB

Apparently my hub load is now severe.

image

Looking at that memory / usage log I would agree, but I can't see it in those device/app status logs, unless I've gone blind and missed something.

2022-11-27 01:34:23,275676,2.33
2022-11-27 01:39:32,277156,2.24
2022-11-27 01:44:42,275064,2.23
2022-11-27 01:49:53,275476,2.48
2022-11-27 01:55:10,276316,2.48
2022-11-27 02:00:24,274880,2.08
2022-11-27 02:05:36,276540,1.85
2022-11-27 02:10:49,275704,1.89
2022-11-27 02:16:08,275392,1.91
2022-11-27 02:21:23,276476,2.04
2022-11-27 02:26:45,275056,2.14
2022-11-27 02:31:58,274720,2.09
2022-11-27 02:37:17,275524,2.15
2022-11-27 02:42:28,276380,2.25
2022-11-27 02:47:49,276096,2.7
2022-11-27 02:52:56,276708,2.53
2022-11-27 02:58:05,277252,3.13
2022-11-27 03:03:16,276380,2.64
2022-11-27 03:08:24,279284,2.18
2022-11-27 03:13:34,276264,2.26
2022-11-27 03:18:55,275972,2.27
2022-11-27 03:24:06,279004,2.29
2022-11-27 03:29:17,281036,1.3
2022-11-27 03:34:21,278832,1.06
2022-11-27 03:39:24,280572,1.2
2022-11-27 03:44:26,284248,0.47
2022-11-27 03:49:28,284384,0.2
2022-11-27 03:54:29,283904,0.3
2022-11-27 03:59:33,281776,0.47
2022-11-27 04:04:38,277520,1.66
2022-11-27 04:09:44,276672,2.07
2022-11-27 04:14:49,276904,2.2
2022-11-27 04:19:55,277264,2.26
2022-11-27 04:25:07,276296,2.65
2022-11-27 04:30:20,274612,2.35
2022-11-27 04:35:30,275156,2.43
2022-11-27 04:40:42,275476,2.46
2022-11-27 04:45:47,276004,2.63
2022-11-27 04:50:54,276628,2.44
2022-11-27 04:56:01,276796,2.33
2022-11-27 05:01:08,276168,2.65
2022-11-27 05:06:24,272892,2.54
2022-11-27 05:11:33,277472,2.87
2022-11-27 05:16:38,275636,2.64
2022-11-27 05:21:44,276152,2.38
2022-11-27 05:26:49,278132,2.11
2022-11-27 05:31:54,275088,1.96
2022-11-27 05:37:00,274692,2.14
2022-11-27 05:42:10,274868,2.83
2022-11-27 05:47:19,274192,2.97
2022-11-27 05:52:40,273232,3.06
2022-11-27 05:57:50,275776,3.25
2022-11-27 06:02:56,274116,3.25
2022-11-27 06:08:02,274132,2.96
2022-11-27 06:13:10,273568,3.14
2022-11-27 06:18:17,273856,3.13
2022-11-27 06:23:32,271892,3.63
2022-11-27 06:28:37,272728,4.24
2022-11-27 06:33:44,272668,3.2
2022-11-27 06:38:50,273852,2.64
2022-11-27 06:43:57,272660,2.54
2022-11-27 06:49:03,273224,2.39
2022-11-27 06:54:10,272584,2.37
2022-11-27 06:59:16,273284,2.41
2022-11-27 07:04:26,271652,2.76
2022-11-27 07:09:39,272644,2.95
2022-11-27 07:14:54,271768,3.08
2022-11-27 07:20:02,271932,2.91
2022-11-27 07:25:17,272116,3.54
2022-11-27 07:30:35,271140,3.59
2022-11-27 07:35:48,271672,3.47
2022-11-27 07:41:03,271708,3.22
2022-11-27 07:46:18,271012,3.39
2022-11-27 07:51:29,271496,3.28
2022-11-27 07:56:39,272128,3.1
2022-11-27 08:01:52,272080,3.23
2022-11-27 08:07:10,270788,3.3
2022-11-27 08:12:23,272832,3.36
2022-11-27 08:17:30,273348,3.08
2022-11-27 08:22:37,273696,2.53
2022-11-27 08:27:44,273672,2.33
2022-11-27 08:32:54,275564,1.93
2022-11-27 08:37:58,274476,1.18
2022-11-27 08:43:02,274152,1.6
2022-11-27 08:48:10,274784,1.29
2022-11-27 08:53:15,273464,1.07
2022-11-27 08:58:19,274036,1.72
2022-11-27 09:03:24,273652,2.04
2022-11-27 09:08:30,275288,2.35
2022-11-27 09:13:35,273540,2.29
2022-11-27 09:18:40,274708,2.25
2022-11-27 09:23:45,273580,2.18
2022-11-27 09:28:50,274076,2.25
2022-11-27 09:33:55,274092,2.34
2022-11-27 09:39:01,273968,2.19
2022-11-27 09:44:08,273628,2.15
2022-11-27 09:49:14,274356,2.15
2022-11-27 09:54:21,274156,2.18
2022-11-27 09:59:27,273712,2.39
2022-11-27 10:04:32,277420,2.13
2022-11-27 10:09:34,279548,0.79
2022-11-27 10:14:39,279856,0.32
2022-11-27 10:19:44,274044,1.34
2022-11-27 10:24:49,272212,2.0
2022-11-27 10:29:54,273308,2.17
2022-11-27 10:35:00,275088,2.18
2022-11-27 10:40:05,272316,2.16
2022-11-27 10:45:11,269992,2.43
2022-11-27 10:50:18,270096,2.49
2022-11-27 10:55:28,269896,3.14
2022-11-27 11:00:41,270720,3.23
2022-11-27 11:06:01,271100,2.64
2022-11-27 11:11:14,270896,2.97
2022-11-27 11:16:27,270548,2.85
2022-11-27 11:21:37,271048,2.83
2022-11-27 11:26:48,270952,3.49
2022-11-27 11:32:00,269932,3.2

All suggestions gratefully received! Otherwise I'll have to drop this one back to v2.3.3. My C-5, which actually does much more than this hub, is not reporting an issue at the moment. My other C-7 is in "elevated" load.

8

Nothing in the logs looks out of the ordinary. Maybe just try rebooting since @jonathanm8 said it resolved it on his. There could just be some back end process that got out of control somehow.

9

First thing I tried, unfortunately. Giving it one more try now. Or maybe I didn't... could have been one of the others. :sweat_smile:

@mike.maxwell - Anything you think I could usefully do for you guys with this hub before I zap it back to v2.3.3? Weird load issue going on (not using HomeKit on this one).

10

Sure give that a try, thats usually the first step when you think its a version issue...

1 Like
11

@mike.maxwell

Seeing the same thing on one of my hubs as well.
I rolled the firmware back to 2.3.3.140 and this appears to have fixed the problem. It has been about 12 hours.
Running 4 hubs. Three are fine (two C5 and one C7)
but the one C3 hub with the problem slowed down over about 2 hours and eventually locks ups.
I had done several soft resets and re-loading different backups with the same results until I rolled back the firmware.

This problem hub I only use for notifications and dashboard. It does not have any zigbee or zwave devices (I have disabled zigbee and removed the zwave stick).
It has Sonos devices, echo skill devices, Ring devices, ecobee, wifi devices, echo devices (via echo speaks), dashboard, and weather. All other devices are connected from my other hubs via the hub mesh.

12

To update, after a few reboots and a couple of nights the hubs all appear to have settled down to normal now. They were becoming unresponsive, to the extent that a button-press would take many seconds to respond, but whatever it was appears to have worked itself out now.

Telling people it's fixed is a guaranteed way to make the problem reappear, so let's see what happens. :wink:

13

@birdslikewires did you roll the firmware back or are you on version 2.3.4.114 ?
Mine is still fine on 2.3.3.140 but may try re-installing version 2.3.4.114.

14

@NoWon - I stayed on 2.3.4.114 and it sorted itself out in the end. Very odd, its load now sits like this:

2022-11-28 22:21:18,454380,0.25
2022-11-28 22:26:20,457436,0.16
2022-11-28 22:31:22,451568,0.11
2022-11-28 22:36:23,450968,0.09
2022-11-28 22:41:25,451328,0.11
2022-11-28 22:46:26,451176,0.13
2022-11-28 22:51:28,451184,0.12
2022-11-28 22:56:29,451388,0.16

Peaceful. Need to give it more to do. :wink:

1 Like
15

Is it possible the load issue was unrelated to the firmware update?

I'm no developer, but why would the new firmware version "settle down" after a period of time?

Either it introduced a change that increased the load on the hub, or it didn't?

1 Like
16

Its certainty possible, also possible it is an intermittent bug causing some back process to get out of control and after a couple of reboots it loads normally (by chance).

17

I saw the same thing on all three of my hubs, two worked themselves out within a few hours while the third was stuck with a load >3.0 for a couple of days. Only occurred after the firmware update. So yes, it could be unrelated, but it would be a hell of a coincidence! :wink:

18

Same one hub was at 7% while not super high is higher than normal. It was not apps or devices as those were each like 2% and .5% according to logs tab. So it must have been an internal process. Didn't reboot as everything was working ok and its now back down to normal 2%

19

@mike.maxwell
I tried updating to C3 hub firmware 2.3.4.114 and 2.3.4.115. I even disabled hub mesh, all apps/rules and all devices (via the left hand check mark). Both times after a few hours the hub locked up on me. I got a data base corrupted error and had to do a soft reset and re-installed backup.
Rolled back to the stable firmware version 2.3.3.140 with everything enabled again.

20

same issue now wither other hub.. is there a way to tell which internal process is doing this..

Screenshot 2022-11-30 165127
Screenshot 2022-11-30 165058
Screenshot 2022-11-30 165042

NO RUnning jobs or pending hub actions

could be this is normal and just reading this thread erronously make me look for it.

21

The CPU % is a snapshot, mine will spike up and down between 2% and 10% since the readings are a single point in time every 5 minutes. The CPU Load tells a better story, and 0.39 is slightly elevated but at times mine gets up to around 0.5. I would watch the load number and see if that is stable or if it continues to climb.

22

CPU % is actually calculated from the 5 minute CPU Load Avg (Load/4 cores*100)

23

Ok, good to know, I just thought you had some secret endpoint with the CPU%, never looked at the code. But anyway, it still will spike up and down constantly, so you would want to track it over time (you can see the recent history using that endpoint I gave above).

image
image1820×771 86.1 KB

24

The weirdness continues. Just now I became unable to turn any of the lights on or off. I check Hub 2 (which runs such things) and discover that after performing fine for a couple of days, the hub load has suddenly jumped, it's reporting "severe" and the load looks like this:

2022-12-01 00:00:46,294324,0.79
2022-12-01 00:05:51,296348,1.04
2022-12-01 00:10:56,295592,1.28
2022-12-01 00:16:01,296372,1.27
2022-12-01 00:21:06,295664,1.18
2022-12-01 00:26:11,295496,1.26
2022-12-01 00:31:16,295252,1.61
2022-12-01 00:36:24,297888,1.11
2022-12-01 00:41:29,296648,1.26
2022-12-01 00:46:34,296776,1.33
2022-12-01 00:51:39,296644,1.26
2022-12-01 00:56:45,296140,1.18
2022-12-01 01:01:50,295704,1.15
2022-12-01 01:06:58,299872,0.86
2022-12-01 01:12:05,300624,0.36
2022-12-01 01:17:07,300448,0.25
2022-12-01 01:22:09,301848,0.22
2022-12-01 01:27:11,302480,0.08
2022-12-01 01:32:16,296252,0.79
2022-12-01 01:37:21,296548,1.11
2022-12-01 01:42:26,296976,1.16
2022-12-01 01:47:31,297704,1.29
2022-12-01 01:52:36,296720,1.65
2022-12-01 01:57:41,295532,1.27
2022-12-01 02:02:46,296144,1.25
2022-12-01 02:07:55,298484,1.19
2022-12-01 02:13:00,297804,1.23
2022-12-01 02:18:05,296928,1.29
2022-12-01 02:23:09,295408,1.31
2022-12-01 02:28:13,297368,1.35
2022-12-01 02:33:18,295824,1.22
2022-12-01 02:38:23,297280,1.16
2022-12-01 02:43:27,296852,1.29
2022-12-01 02:48:32,296548,1.25
2022-12-01 02:53:38,297152,1.16
2022-12-01 02:58:43,296316,1.14
2022-12-01 03:03:49,294592,1.29
2022-12-01 03:08:54,297088,1.16
2022-12-01 03:13:58,296592,1.13
2022-12-01 03:19:04,297156,1.16
2022-12-01 03:24:09,294188,1.17
2022-12-01 03:29:15,295464,1.25
2022-12-01 03:34:20,295804,1.42
2022-12-01 03:39:26,295480,1.19
2022-12-01 03:44:31,296324,1.15
2022-12-01 03:49:37,295416,1.11
2022-12-01 03:54:42,297220,1.17
2022-12-01 03:59:48,296528,1.07
2022-12-01 04:04:53,295256,1.13
2022-12-01 04:09:59,296472,1.19
2022-12-01 04:15:05,287960,1.29
2022-12-01 04:20:09,296384,1.17
2022-12-01 04:25:12,296392,1.1
2022-12-01 04:30:21,291808,1.09
2022-12-01 04:35:25,294216,1.25
2022-12-01 04:40:28,293804,1.14
2022-12-01 04:45:32,293976,1.16
2022-12-01 04:50:37,295384,1.23
2022-12-01 04:55:41,294004,1.16
2022-12-01 05:00:45,294064,1.07
2022-12-01 05:05:49,292532,1.08
2022-12-01 05:10:54,293112,1.16
2022-12-01 05:15:59,294892,1.2
2022-12-01 05:21:03,293496,1.25
2022-12-01 05:26:08,295188,1.2
2022-12-01 05:31:12,294092,1.23
2022-12-01 05:36:17,293440,1.15
2022-12-01 05:41:20,295132,1.14
2022-12-01 05:46:25,294544,1.41
2022-12-01 05:51:28,295552,1.29
2022-12-01 05:56:32,294268,1.28
2022-12-01 06:01:37,294300,1.21
2022-12-01 06:06:42,293776,1.21
2022-12-01 06:11:47,293532,1.21
2022-12-01 06:16:52,295264,1.1
2022-12-01 06:21:57,294132,1.11
2022-12-01 06:27:01,292368,1.2
2022-12-01 06:32:06,294396,1.23
2022-12-01 06:37:10,294040,1.23
2022-12-01 06:42:15,292888,1.13
2022-12-01 06:47:19,293980,1.19
2022-12-01 06:52:24,293264,1.12
2022-12-01 06:57:27,293668,1.17
2022-12-01 07:02:32,292440,1.14
2022-12-01 07:07:35,293724,1.19
2022-12-01 07:12:39,293724,1.13
2022-12-01 07:17:47,293140,1.13
2022-12-01 07:22:52,294576,1.04
2022-12-01 07:27:56,292200,1.06
2022-12-01 07:33:00,293020,1.0
2022-12-01 07:38:04,293672,1.17
2022-12-01 07:43:08,293024,1.44
2022-12-01 07:48:12,294028,1.24
2022-12-01 07:53:15,293156,1.27
2022-12-01 07:58:20,293592,1.23
2022-12-01 08:03:23,293624,1.31
2022-12-01 08:08:27,293488,1.23
2022-12-01 08:13:32,293472,1.27
2022-12-01 08:18:35,293164,1.25
2022-12-01 08:23:40,294344,1.19
2022-12-01 08:28:45,293596,1.12
2022-12-01 08:33:49,292904,1.2
2022-12-01 08:38:56,293068,1.14
2022-12-01 08:44:01,291872,1.27
2022-12-01 08:49:06,291784,1.25
2022-12-01 08:54:11,293464,1.22
2022-12-01 08:59:18,292596,1.14
2022-12-01 09:04:21,297160,0.98
2022-12-01 09:09:29,295888,0.39
2022-12-01 09:14:33,292732,0.96
2022-12-01 09:19:38,293340,1.15
2022-12-01 09:24:42,291948,1.1
2022-12-01 09:29:47,293156,1.12
2022-12-01 09:34:51,293672,1.13
2022-12-01 09:39:56,293276,1.12
2022-12-01 09:44:59,294676,1.27
2022-12-01 09:50:07,294888,1.08
2022-12-01 09:55:10,292264,1.3
2022-12-01 10:00:14,289092,1.4
2022-12-01 10:05:19,292912,1.28
2022-12-01 10:10:22,293312,1.18
2022-12-01 10:15:27,293176,1.2
2022-12-01 10:20:31,291188,1.23
2022-12-01 10:25:36,293060,1.15
2022-12-01 10:30:41,293436,1.2
2022-12-01 10:35:45,293472,1.19
2022-12-01 10:40:50,291716,1.13
2022-12-01 10:45:55,293160,1.26
2022-12-01 10:50:58,291836,1.33
2022-12-01 10:56:03,292888,1.19
2022-12-01 11:01:09,292128,1.25
2022-12-01 11:06:18,295372,1.2
2022-12-01 11:11:23,292888,1.17
2022-12-01 11:16:28,293716,1.28
2022-12-01 11:21:34,292712,1.19
2022-12-01 11:26:39,294120,1.18
2022-12-01 11:31:44,293016,1.24
2022-12-01 11:36:48,292496,1.25
2022-12-01 11:41:53,293148,1.35
2022-12-01 11:46:58,292032,1.25
2022-12-01 11:52:02,296436,1.15
2022-12-01 11:57:08,290896,1.13
2022-12-01 12:02:13,291580,1.11
2022-12-01 12:07:22,294556,1.17
2022-12-01 12:12:27,291672,1.24
2022-12-01 12:17:33,290948,1.27
2022-12-01 12:22:38,291984,1.57
2022-12-01 12:27:43,292464,1.37
2022-12-01 12:32:47,293848,1.26
2022-12-01 12:37:53,292656,1.26
2022-12-01 12:42:57,297232,0.73
2022-12-01 12:48:01,293492,0.43
2022-12-01 12:53:08,293376,0.78
2022-12-01 12:58:12,292904,1.09
2022-12-01 13:03:15,292380,1.51
2022-12-01 13:08:20,293568,1.23
2022-12-01 13:13:24,292568,1.29
2022-12-01 13:18:28,290964,1.18
2022-12-01 13:23:32,292748,1.14
2022-12-01 13:28:36,293980,1.12
2022-12-01 13:33:41,294440,1.17
2022-12-01 13:38:45,292804,1.24
2022-12-01 13:43:48,292980,1.22
2022-12-01 13:48:52,294140,1.18
2022-12-01 13:53:57,293556,1.18
2022-12-01 13:59:02,293792,1.21
2022-12-01 14:04:06,294144,1.16
2022-12-01 14:09:11,292732,1.14
2022-12-01 14:14:16,294008,1.09
2022-12-01 14:19:21,293520,1.16
2022-12-01 14:24:26,293892,1.17
2022-12-01 14:29:33,293892,1.16
2022-12-01 14:34:41,293868,1.15
2022-12-01 14:39:48,294988,1.22
2022-12-01 14:44:56,292856,1.14
2022-12-01 14:50:03,294180,1.13
2022-12-01 14:55:09,292388,1.23
2022-12-01 15:00:15,290380,1.25
2022-12-01 15:05:23,293584,1.3
2022-12-01 15:10:30,290248,1.3
2022-12-01 15:15:38,292892,1.2
2022-12-01 15:20:45,290216,1.25
2022-12-01 15:25:52,291720,1.24
2022-12-01 15:30:58,290988,1.21
2022-12-01 15:36:05,291164,1.16
2022-12-01 15:41:12,290988,1.24
2022-12-01 15:46:18,291856,1.19
2022-12-01 15:51:26,292068,1.21
2022-12-01 15:56:34,291216,1.21
2022-12-01 16:01:42,291604,1.35
2022-12-01 16:06:49,291496,1.28
2022-12-01 16:11:56,291156,1.19
2022-12-01 16:17:04,292140,1.19
2022-12-01 16:22:11,291420,1.26
2022-12-01 16:27:20,292132,1.16
2022-12-01 16:32:27,290588,1.24
2022-12-01 16:37:36,291140,1.26
2022-12-01 16:42:44,292164,1.33
2022-12-01 16:47:52,289604,1.22
2022-12-01 16:52:59,292588,1.29
2022-12-01 16:58:06,290068,1.18
2022-12-01 17:03:14,291812,1.34
2022-12-01 17:08:23,292240,1.21
2022-12-01 17:13:29,290984,1.14
2022-12-01 17:18:36,290896,1.11
2022-12-01 17:23:44,291512,1.3
2022-12-01 17:28:52,291244,1.35
2022-12-01 17:34:00,290916,1.35
2022-12-01 17:39:07,290412,1.21
2022-12-01 17:44:20,291676,1.2
2022-12-01 17:49:28,291852,1.37
2022-12-01 17:54:34,290272,1.21
2022-12-01 17:59:43,291600,1.22
2022-12-01 18:04:51,290204,1.18
2022-12-01 18:09:59,290944,1.31
2022-12-01 18:15:07,283720,1.26
2022-12-01 18:20:14,289128,1.24
2022-12-01 18:25:22,290852,1.43
2022-12-01 18:30:33,285628,1.6
2022-12-01 18:35:41,292492,1.38
2022-12-01 18:40:49,288732,1.28
2022-12-01 18:45:57,290608,1.37
2022-12-01 18:51:05,289876,1.38
2022-12-01 18:56:14,289240,1.31
2022-12-01 19:01:31,289512,1.51
2022-12-01 19:06:42,289884,1.65
2022-12-01 19:11:58,288908,1.95
2022-12-01 19:17:12,289836,2.0
2022-12-01 19:22:19,288420,2.02
2022-12-01 19:27:34,290048,2.08
2022-12-01 19:32:45,289784,2.01
2022-12-01 19:37:53,288020,1.88
2022-12-01 19:43:03,289224,1.98
2022-12-01 19:48:17,288128,1.87
2022-12-01 19:53:32,288160,2.01
2022-12-01 19:58:43,288052,1.9
2022-12-01 20:03:54,289184,1.94
2022-12-01 20:09:09,287848,1.99
2022-12-01 20:14:16,288636,1.86
2022-12-01 20:19:25,288356,1.88
2022-12-01 20:24:34,288264,1.95
2022-12-01 20:29:46,287720,1.91
2022-12-01 20:34:59,287968,1.99
2022-12-01 20:40:10,286912,1.96
2022-12-01 20:45:28,280992,2.14
2022-12-01 20:50:34,284772,2.17
2022-12-01 20:55:43,285876,2.13
2022-12-01 21:00:53,282092,1.93

Scrolling to the bottom of that will show hub loads spiking above 2.00 this evening. We've done nothing different tonight from any preceding night.

I might try the new .116 release, but I don't see any mention of these things in there. :confused:

25

if it was at 2am likely it was doing a backup.

26

Nope, it's been from about 1900 to now, hovering around that 2.00 load level. It's responding again at the moment, but sluggishly.