Daily issue with severe hub load

🛎️ Get Help
1

Hi all, since I've purchased my C-8 (about 2 months ago now), I've been battling a fairly consistent issue with the hub reporting "Hub load is [severe]" and I've been unable to find a cause/resolution. I unfortunately do not have any previous data to compare against with an older model, as this is my first Hubitat.

My device info:

  • Model: C-8
  • Platform Version: 2.3.5.146
  • LAN Connection: Hardwired Ethernet

I've been combing the forums for an answer and have seen similar reports of this issue (a few below); however, the other reports have either also been unsolved or their resolution does not correlate to my issue.

Hub Load Severe
[.130] Severe hub load and more
Elevated and Severe Hub Loads with v2.3.4 - #356 by mavrrick58

The general behavior I'm seeing is as follows: the hub will be running normally for approx. 24 hours and then the hub will report "Hub load is [severe]" and will stay in this state until I reboot the hub. After a reboot, everything settles back down and within about 24 hours the severe load cycle repeats. To note, when the hub reports severe load, my automatons all still seem to work; however, my notifications via Pushover are definitely delayed and the console UI lags a decent amount.

I've looked through the "Past Logs" and Device and App stats, but the "% of busy" numbers do not seem to be bad, unless I'm misunderstanding the data, which is quite possible. Below are my current numbers with the Hub in a "severe" load state. Also, I did perform a Soft Reset (and restore) on the device hoping that would help, but unfortunately it did not.

If anyone could possibly offer an insight into what might be causing this, I'd greatly appreciate it!

Note - The "Alarm System" device is using the QolSys IQ Alarm Panel Drivers in case it's helpful

Hub_Info_Driver

App_Stats
App_Stats2147Ă—744 62.1 KB

Device_Stats
Device_Stats2155Ă—824 63.6 KB

Location_Events
Location_Events1857Ă—880 63.9 KB

1 Like
2

The problem with looking at app/device stats is that it only shows a very specific data related to time spent runing code sections. There are many other kind of activity on the hub that you may not see.

One such activity is tcp activity to create connections to lan/wifi devices or external services. I suspect something to do with how your alarm system is setup is causing the load.

There is no doubt your hub is busy as a load of 4 is high and the temp of 120 on a C8 shows the cpu is being hit hard.

You may want try cutting back on what you have running on the hub to see if you can get it to stabilize. Then add stuff back.

Personally I have seen these two things happen.

  1. A external service using Maker API that made one call every 12 seconds to update a device brought my dev hub to its knees.
  2. Bad code create a race condition because of bad timing. That over time caused the hub to overload.
1 Like
3

My suggestion is to start eliminating app/rules by disabling or pausing them and see if behavior changes. I'd start with Alarm System. As Mavrrick58 suggested frequent TCP activity can be a cause, and I've had the same experience with 3rd party wifi devices(though not with officially supported devices, like Kasa plugs/lights).
When the hub slows what are your memory stats at that point?
http://10.10.20.10/hub/advanced/freeOSMemoryHistory

As a temporary workaround you could reboot every 18 hours via settings in Hub Info Driver. I think there's a setting that says reboot when mem <xxx

1 Like
4

@mavrrick58 and @Rxich,
Thanks so much for the suggestions. I'm going to try Disabling the "Alarm System" device first to see if that is the culprit, and I'll report back with results.

Below is a copy of my memory stats around when the Hub CPU spikes up to current time (I have not rebooted yet since it spiked on 6/13). It does look like once the spike occurs the Free OS Memory keeps dropping, almost like there is a Memory Leak somewhere.

Date/time Free OS 5m CPU avg
6/13/2023 16:11 522500 0.1
6/13/2023 16:16 521656 0.11
6/13/2023 16:21 521704 0.12
6/13/2023 16:26 521800 0.1
6/13/2023 16:31 522068 0.24
6/13/2023 16:36 521672 0.13
6/13/2023 16:41 521992 0.15
6/13/2023 16:46 522208 0.1
6/13/2023 16:51 521992 0.08
6/13/2023 16:56 522300 0.08
6/13/2023 17:01 521772 0.11
6/13/2023 17:06 521912 0.09
6/13/2023 17:11 521480 0.08
6/13/2023 17:17 520656 0.22
6/13/2023 17:22 521076 0.12
6/13/2023 17:27 521772 0.15
6/13/2023 17:32 522136 0.07
6/13/2023 17:37 521444 0.11
6/13/2023 17:42 521144 0.1
6/13/2023 17:47 519804 0.05
6/13/2023 17:52 520256 0.16
6/13/2023 17:57 520472 0.84
6/13/2023 18:02 519880 2.93
6/13/2023 18:07 520064 3.7
6/13/2023 18:12 519708 3.92
6/13/2023 18:18 520572 4.09
6/13/2023 18:23 519412 4.05
6/13/2023 18:28 518684 4.06
6/13/2023 18:33 519036 4.12
6/13/2023 18:38 519052 4.18
6/13/2023 18:43 520076 4.08
6/13/2023 18:48 519040 4.14
6/13/2023 18:53 519148 4.19
6/13/2023 18:58 519564 4.07
6/13/2023 19:03 518912 4.17
6/13/2023 19:08 519052 4.07
6/13/2023 19:14 518988 4.04
6/13/2023 19:19 519100 4.09
6/13/2023 19:24 519224 4.14
6/13/2023 19:29 519456 4.06
6/13/2023 19:34 518852 4.06
6/13/2023 19:39 518932 4.03
6/13/2023 19:44 518700 4.03
6/13/2023 19:49 519336 4.09
6/13/2023 19:54 518836 4.06
6/13/2023 19:59 519176 4.03
6/13/2023 20:04 519672 4.14
6/13/2023 20:09 519848 4.06
6/13/2023 20:15 518488 4.08
6/13/2023 20:20 519128 4.09
6/13/2023 20:25 518564 4.09
6/13/2023 20:30 516472 4.12
6/13/2023 20:35 516952 4.13
6/13/2023 20:40 517600 4.14
6/13/2023 20:45 516668 4.23
6/13/2023 20:50 517056 4.11
6/13/2023 20:55 516092 4.15
6/13/2023 21:00 515876 4.15
6/13/2023 21:05 516480 4.13
6/13/2023 21:11 516420 4.05
6/13/2023 21:16 516984 4.12
6/13/2023 21:21 517472 4.1
6/13/2023 21:26 513568 4.11
6/13/2023 21:31 515080 4.09
6/13/2023 21:36 515532 4.04
6/13/2023 21:41 515460 4
6/13/2023 21:46 517136 4.03
6/13/2023 21:51 516796 4
6/13/2023 21:56 516548 4.02
6/13/2023 22:01 516116 4.06
6/13/2023 22:06 515508 4.12
6/13/2023 22:12 515108 4.08
6/13/2023 22:17 515492 4.08
6/13/2023 22:22 516424 4.04
6/13/2023 22:27 515664 4.01
6/13/2023 22:32 515692 4.09
6/13/2023 22:37 514808 4.03
6/13/2023 22:42 514888 4
6/13/2023 22:47 515800 4.05
6/13/2023 22:52 516156 4.03
6/13/2023 22:57 515004 4.01
6/13/2023 23:02 515052 4.03
6/13/2023 23:08 514044 4.06
6/13/2023 23:13 514012 4.04
6/13/2023 23:18 514052 4.07
6/13/2023 23:23 514268 4.01
6/13/2023 23:28 513952 4
6/13/2023 23:33 513984 4
6/13/2023 23:38 513920 4
6/13/2023 23:43 513316 4.04
6/13/2023 23:48 514156 4.04
6/13/2023 23:53 513264 4.02
6/13/2023 23:58 512816 4
6/14/2023 0:04 510680 4.05
6/14/2023 0:09 510916 4.06
6/14/2023 0:14 511328 4.03
6/14/2023 0:19 511392 4.01
6/14/2023 0:24 510696 4
6/14/2023 0:29 510740 4
6/14/2023 0:34 510832 4.04
6/14/2023 0:39 510528 4
6/14/2023 0:44 510796 4
6/14/2023 0:49 511072 4
6/14/2023 0:54 510884 4.02
6/14/2023 0:59 510888 4
6/14/2023 1:05 510760 4
6/14/2023 1:10 510936 4.03
6/14/2023 1:15 510156 4.05
6/14/2023 1:20 509752 4.02
6/14/2023 1:25 509720 4.01
6/14/2023 1:30 510632 4
6/14/2023 1:35 510516 4
6/14/2023 1:40 509696 4
6/14/2023 1:45 509584 4.02
6/14/2023 1:50 510152 4.03
6/14/2023 1:55 509660 4.01
6/14/2023 2:01 508776 4.03
6/14/2023 2:06 509452 4
6/14/2023 2:11 509144 4.05
6/14/2023 2:16 509360 4.07
6/14/2023 2:21 510528 4.01
6/14/2023 2:26 509640 4
6/14/2023 2:31 509392 4.04
6/14/2023 2:36 509560 4.02
6/14/2023 2:41 509548 4
6/14/2023 2:46 508140 4.02
6/14/2023 2:51 508000 4
6/14/2023 2:57 508604 4
6/14/2023 3:02 503116 4.25
6/14/2023 3:07 505808 4.09
6/14/2023 3:12 505428 4.02
6/14/2023 3:17 507008 4.09
6/14/2023 3:22 506424 4.02
6/14/2023 3:27 506064 4.06
6/14/2023 3:32 505808 4.08
6/14/2023 3:37 506116 4.02
6/14/2023 3:42 506172 4
6/14/2023 3:47 506236 4.02
6/14/2023 3:52 505616 4.02
6/14/2023 3:58 504576 4.05
6/14/2023 4:03 504448 4.09
6/14/2023 4:08 504772 4.04
6/14/2023 4:13 504300 4.02
6/14/2023 4:18 505044 4.06
6/14/2023 4:23 505684 4.04
6/14/2023 4:28 504408 4.06
6/14/2023 4:33 504612 4.04
6/14/2023 4:38 504192 4.06
6/14/2023 4:43 504564 4.04
6/14/2023 4:48 504176 4.06
6/14/2023 4:54 503912 4.08
6/14/2023 4:59 504352 4.09
6/14/2023 5:04 504736 4.13
6/14/2023 5:09 505328 4.06
6/14/2023 5:14 504132 4.02
6/14/2023 5:19 505112 4.05
6/14/2023 5:24 505096 4.1
6/14/2023 5:29 504896 4.04
6/14/2023 5:34 503796 4.16
6/14/2023 5:39 503880 4.05
6/14/2023 5:44 504388 4.08
6/14/2023 5:50 504576 4.05
6/14/2023 5:55 504436 4.06
6/14/2023 6:00 505280 4.02
6/14/2023 6:05 505088 4.03
6/14/2023 6:10 504528 4
6/14/2023 6:15 504928 4.12
6/14/2023 6:20 505088 4.06
6/14/2023 6:25 504948 4.18
6/14/2023 6:30 504236 4.13
6/14/2023 6:35 505120 4.1
6/14/2023 6:40 505312 4.02
6/14/2023 6:46 504876 4.1
6/14/2023 6:51 504740 4.02
6/14/2023 6:56 504400 4.04
6/14/2023 7:01 505764 4.04
6/14/2023 7:06 505036 4.04
6/14/2023 7:11 504664 4.06
6/14/2023 7:16 503792 4.05
6/14/2023 7:21 504016 4.01
6/14/2023 7:26 504208 4.04
6/14/2023 7:31 504144 4.13
6/14/2023 7:36 504472 4.08
6/14/2023 7:41 504292 4.06
6/14/2023 7:47 504344 4.15
6/14/2023 7:52 504236 4.05
6/14/2023 7:57 503848 4.04
6/14/2023 8:02 504316 4.04
6/14/2023 8:07 504316 4.14
6/14/2023 8:12 504164 4.08
6/14/2023 8:17 504736 4.06
6/14/2023 8:22 504772 4.1
6/14/2023 8:27 504164 4.09
6/14/2023 8:32 504040 4.06
6/14/2023 8:37 503956 4.05
6/14/2023 8:43 504068 4.01
6/14/2023 8:48 504052 4.07
6/14/2023 8:53 504284 4.02
6/14/2023 8:58 503028 4.03
6/14/2023 9:03 503164 4.03
6/14/2023 9:08 503152 4.03
6/14/2023 9:13 502760 4.06
6/14/2023 9:18 502436 4.03
6/14/2023 9:23 502744 4.03
6/14/2023 9:28 503600 4.02
6/14/2023 9:33 503276 4.03
6/14/2023 9:39 502816 4.03
6/14/2023 9:44 502608 4.01
6/14/2023 9:49 502624 4.03
6/14/2023 9:54 502252 4.06
6/14/2023 9:59 502156 4.03
6/14/2023 10:04 501936 4.01
6/14/2023 10:09 502088 4.01
6/14/2023 10:14 501872 4.18
6/14/2023 10:19 503128 4.1
6/14/2023 10:24 502580 4.05
6/14/2023 10:29 502800 4.03
6/14/2023 10:34 502488 4.02
6/14/2023 10:40 502400 4.04
6/14/2023 10:45 502192 4.02
6/14/2023 10:50 502756 4.01
6/14/2023 10:55 501776 4.1
6/14/2023 11:00 501048 4.15
6/14/2023 11:05 501576 4.11
6/14/2023 11:10 501420 4.04
6/14/2023 11:15 500624 4.09
6/14/2023 11:20 501344 4.02
6/14/2023 11:25 501804 4.11
6/14/2023 11:30 501076 4.07
6/14/2023 11:36 501508 4.03
6/14/2023 11:41 501336 4.01
6/14/2023 11:46 501152 4.02
6/14/2023 11:51 501188 4.01
6/14/2023 11:56 500820 4.03
6/14/2023 12:01 501016 4.07
6/14/2023 12:06 500704 4.1
6/14/2023 12:11 500848 4.14
6/14/2023 12:16 499680 4.14
6/14/2023 12:21 500032 4.1
6/14/2023 12:26 501172 4.04
6/14/2023 12:31 500368 4.02
6/14/2023 12:37 500508 4.08
6/14/2023 12:42 500368 4.1
6/14/2023 12:47 500840 4.1
6/14/2023 12:52 500780 4.04
6/14/2023 12:57 500048 4.1
6/14/2023 13:02 500156 4.05
6/14/2023 13:07 499688 4.04
6/14/2023 13:12 499704 4.09
6/14/2023 13:17 501292 4.06
6/14/2023 13:22 500976 4.05
6/14/2023 13:27 500760 4.06
6/14/2023 13:33 501100 4.06
6/14/2023 13:38 500388 4.02
6/14/2023 13:43 500984 4.05
6/14/2023 13:48 500580 4.03
6/14/2023 13:53 501544 4.04
6/14/2023 13:58 500640 4.05
6/14/2023 14:03 501608 4.09
6/14/2023 14:08 501500 4.1
6/14/2023 14:13 500704 4.1
6/14/2023 14:18 500888 4.05
6/14/2023 14:23 501364 4.02
6/14/2023 14:29 501084 4.01
6/14/2023 14:34 501084 4.02
6/14/2023 14:39 501256 4.03
6/14/2023 14:44 500260 4.01
6/14/2023 14:49 500252 4
6/14/2023 14:54 499672 4.03
6/14/2023 14:59 500480 4.01
6/14/2023 15:04 499720 4.03
6/14/2023 15:09 499992 4
6/14/2023 15:14 498052 4.02
6/14/2023 15:19 499848 4
6/14/2023 15:24 500088 4.04
6/14/2023 15:30 497344 4
6/14/2023 15:35 500420 4.16
6/14/2023 15:40 500232 4.05
6/14/2023 15:45 500136 4.14
6/14/2023 15:50 499580 4.07
6/14/2023 15:55 499516 4.09
6/14/2023 16:00 499204 4.07
6/14/2023 16:05 499512 4.11
6/14/2023 16:10 499220 4.08
6/14/2023 16:15 498596 4.05
6/14/2023 16:20 498844 4.01
6/14/2023 16:26 498688 4.07
6/14/2023 16:31 499172 4.11
6/14/2023 16:36 498636 4.12
6/14/2023 16:41 498620 4.06
6/14/2023 16:46 499536 4.06
6/14/2023 16:51 499088 4.04
6/14/2023 16:56 498768 4.04
6/14/2023 17:01 498672 4.11
6/14/2023 17:06 498800 4.08
6/14/2023 17:11 498188 4.11
6/14/2023 17:16 498120 4.03
6/14/2023 17:21 498156 4.06
6/14/2023 17:27 496024 4.04
6/14/2023 17:32 497524 4.05
6/14/2023 17:37 496828 4.03
6/14/2023 17:42 497520 4
6/14/2023 17:47 497116 4.07
6/14/2023 17:52 496436 4.05
6/14/2023 17:57 496068 4.02
6/14/2023 18:02 496056 4.01
6/14/2023 18:07 496400 4
6/14/2023 18:12 496324 4
6/14/2023 18:17 497148 4.04
6/14/2023 18:23 497092 4
6/14/2023 18:28 496596 4
6/14/2023 18:33 496736 4.01
6/14/2023 18:38 496604 4.01
6/14/2023 18:43 496492 4.03
6/14/2023 18:48 496128 4.05
6/14/2023 18:53 496324 4.05
6/14/2023 18:58 495912 4.01
6/14/2023 19:03 495624 4
6/14/2023 19:08 495420 4
6/14/2023 19:13 495604 4.01
6/14/2023 19:18 496068 4.03
6/14/2023 19:24 495964 4.08
6/14/2023 19:29 495228 4.04
6/14/2023 19:34 495340 4.05
6/14/2023 19:39 494988 4.04
6/14/2023 19:44 495240 4.02
6/14/2023 19:49 494880 4.03
6/14/2023 19:54 494796 4.01
6/14/2023 19:59 495048 4.03
6/14/2023 20:04 494840 4.05
6/14/2023 20:09 495200 4.13
6/14/2023 20:14 494284 4.07
6/14/2023 20:20 495696 4.11
6/14/2023 20:25 495720 4.11
6/14/2023 20:30 494652 4.15
6/14/2023 20:35 495848 4.08
6/14/2023 20:40 495772 4.12
6/14/2023 20:45 496024 4.09
6/14/2023 20:50 496228 4.04
6/14/2023 20:55 495712 4.02
6/14/2023 21:00 495868 4.04
6/14/2023 21:05 495356 4.02
6/14/2023 21:10 495480 4.04
6/14/2023 21:16 494792 4.04
6/14/2023 21:21 495100 4.05
6/14/2023 21:26 495004 4.07
6/14/2023 21:31 494840 4.06
6/14/2023 21:36 494940 4.15
6/14/2023 21:41 494468 4.2
6/14/2023 21:46 494500 4.16
6/14/2023 21:51 494556 4.08
6/14/2023 21:56 494160 4.07
6/14/2023 22:01 493948 4.07
6/14/2023 22:06 493916 4.02
6/14/2023 22:11 493712 4.03
6/14/2023 22:17 493788 4.04
6/14/2023 22:22 494328 4.02
6/14/2023 22:27 493628 4
6/14/2023 22:32 494256 4.01
6/14/2023 22:37 493740 4
6/14/2023 22:42 493976 4.15
6/14/2023 22:47 493012 4.14
6/14/2023 22:52 493820 4.15
6/14/2023 22:57 493012 4.06
6/14/2023 23:02 493432 4.09
6/14/2023 23:07 493228 4.07
6/14/2023 23:13 493092 4.01
6/14/2023 23:18 493076 4.07
6/14/2023 23:23 493140 4.14
6/14/2023 23:28 494012 4.08
6/14/2023 23:33 493052 4.04
6/14/2023 23:38 493276 4.05
6/14/2023 23:43 493012 4.07
6/14/2023 23:48 494252 4.03
6/14/2023 23:53 494192 4
6/14/2023 23:58 493864 4.11
6/15/2023 0:03 493964 4.07
6/15/2023 0:09 493608 4.11
6/15/2023 0:14 492848 4.06
6/15/2023 0:19 493504 4.14
6/15/2023 0:24 493428 4.08
6/15/2023 0:29 493036 4.08
6/15/2023 0:34 494264 4.04
6/15/2023 0:39 493780 4.05
6/15/2023 0:44 493548 4.03
6/15/2023 0:49 493592 4.03
6/15/2023 0:54 493092 4.02
6/15/2023 0:59 493356 4
6/15/2023 1:04 492340 4.06
6/15/2023 1:10 492076 4.08
6/15/2023 1:15 492132 4.08
6/15/2023 1:20 493036 4.02
6/15/2023 1:25 492648 4.04
6/15/2023 1:30 493504 4.14
6/15/2023 1:35 493564 4.1
6/15/2023 1:40 493300 4.02
6/15/2023 1:45 492496 4
6/15/2023 1:50 492968 4
6/15/2023 1:55 492452 4.02
6/15/2023 2:00 492700 4.02
6/15/2023 2:06 492636 4.01
6/15/2023 2:11 492748 4.06
6/15/2023 2:16 491644 4.13
6/15/2023 2:21 492000 4.05
6/15/2023 2:26 492464 4.04
6/15/2023 2:31 491972 4.1
6/15/2023 2:36 491984 4.1
6/15/2023 2:41 491504 4.11
6/15/2023 2:46 492216 4.16
6/15/2023 2:51 492360 4.16
6/15/2023 2:56 492048 4.1
6/15/2023 3:02 482992 4.32
6/15/2023 3:07 490140 4.2
6/15/2023 3:12 491172 4.1
6/15/2023 3:17 490584 4.06
6/15/2023 3:22 490704 4.05
6/15/2023 3:27 490404 4.03
6/15/2023 3:32 490264 4.02
6/15/2023 3:37 488600 4.03
6/15/2023 3:42 490412 4.01
6/15/2023 3:47 490316 4.01
6/15/2023 3:52 490896 4.06
6/15/2023 3:58 490972 4.12
6/15/2023 4:03 491304 4.08
6/15/2023 4:08 491992 4.1
6/15/2023 4:13 492024 4.06
6/15/2023 4:18 491340 4.04
6/15/2023 4:23 491420 4.01
6/15/2023 4:28 492152 4
6/15/2023 4:33 492132 4.12
6/15/2023 4:38 490980 4.04
6/15/2023 4:43 490904 4.02
6/15/2023 4:48 490792 4
6/15/2023 4:53 490948 4
6/15/2023 4:59 490436 4
6/15/2023 5:04 490888 4
6/15/2023 5:09 490052 4
6/15/2023 5:14 490316 4
6/15/2023 5:19 490052 4
6/15/2023 5:24 490680 4.01
6/15/2023 5:29 490452 4.01
6/15/2023 5:34 490592 4.02
6/15/2023 5:39 489988 4.1
6/15/2023 5:44 489752 4.04
6/15/2023 5:49 490388 4.03
6/15/2023 5:55 489844 4.04
6/15/2023 6:00 489196 4.06
6/15/2023 6:05 489536 4.06
6/15/2023 6:10 489780 4.01
6/15/2023 6:15 490388 4.01
6/15/2023 6:20 490256 4
6/15/2023 6:25 489380 4
6/15/2023 6:30 489320 4.02
6/15/2023 6:35 489320 4.02
6/15/2023 6:40 489228 4
6/15/2023 6:45 488792 4.06
6/15/2023 6:50 488952 4.01
6/15/2023 6:56 488124 4
6/15/2023 7:01 488460 4.04
6/15/2023 7:06 488148 4.1
6/15/2023 7:11 488584 4.04
6/15/2023 7:16 488272 4.03
6/15/2023 7:21 488928 4
6/15/2023 7:26 488088 4.02
6/15/2023 7:31 488352 4.04
6/15/2023 7:36 488052 4
6/15/2023 7:41 488708 4
6/15/2023 7:46 489172 4.03
6/15/2023 7:52 488496 4.02
6/15/2023 7:57 488328 4
6/15/2023 8:02 488080 4.09
6/15/2023 8:07 488208 4.04
6/15/2023 8:12 488656 4.02
6/15/2023 8:17 488232 4.04
6/15/2023 8:22 488648 4.02
6/15/2023 8:27 488380 4.1
6/15/2023 8:32 488292 4.08
6/15/2023 8:37 488284 4.1
6/15/2023 8:42 488068 4.06
6/15/2023 8:47 487892 4.09
6/15/2023 8:53 487676 4.04
6/15/2023 8:58 487644 4.06
6/15/2023 9:03 488340 4.04
6/15/2023 9:08 488064 4.05
6/15/2023 9:13 487548 4.1
6/15/2023 9:18 488152 4.09
6/15/2023 9:28 488616 4.09
6/15/2023 9:33 488384 4.1
6/15/2023 9:38 487968 4.11
6/15/2023 9:43 488284 4.03
6/15/2023 9:48 489040 4.06
6/15/2023 9:53 489108 4.05
6/15/2023 9:59 488592 4.04
6/15/2023 10:04 488796 4.02
6/15/2023 10:09 488436 4.05
6/15/2023 10:14 487852 4.14
6/15/2023 10:19 488240 4.09
6/15/2023 10:24 488256 4.03
6/15/2023 10:29 487816 4.03
6/15/2023 10:34 487960 4.01
6/15/2023 10:39 487772 4.01
6/15/2023 10:44 487804 4
6/15/2023 10:49 487992 4.03
6/15/2023 10:55 488272 4
6/15/2023 11:00 487640 4.06
6/15/2023 11:05 486732 4.17
6/15/2023 11:10 487104 4.09
6/15/2023 11:15 486396 4.27
6/15/2023 11:20 487072 4.1
6/15/2023 11:25 486176 4.21
6/15/2023 11:30 487092 4.2
6/15/2023 11:35 486952 4.27
6/15/2023 11:40 486988 4.1
6/15/2023 11:45 486516 4.23
6/15/2023 11:51 486404 4.09
6/15/2023 11:56 486300 4.21
6/15/2023 12:01 486720 4.1
6/15/2023 12:06 486564 4.08
6/15/2023 12:11 485980 4.08
6/15/2023 12:16 485212 4.17
6/15/2023 12:21 485692 4.07
6/15/2023 12:26 485392 4.04
6/15/2023 12:31 485008 4.05
6/15/2023 12:36 485592 4.04
6/15/2023 12:41 485936 4.02
6/15/2023 12:46 485408 4.05
6/15/2023 12:52 485272 4.02
6/15/2023 12:57 485004 4.06
6/15/2023 13:02 485052 4.07
6/15/2023 13:07 485004 4.06
6/15/2023 13:12 484680 4.15
6/15/2023 13:17 484276 4.15
6/15/2023 13:22 484356 4.07
6/15/2023 13:27 484024 4.08
6/15/2023 13:32 483900 4.11
6/15/2023 13:37 484068 4.09
6/15/2023 13:42 484196 4.04
6/15/2023 13:48 484020 4.06
6/15/2023 13:53 485372 4.02
6/15/2023 13:58 483756 4.02
6/15/2023 14:03 483236 4.01
6/15/2023 14:08 483336 4
6/15/2023 14:13 483208 4
6/15/2023 14:18 484264 4.16
6/15/2023 14:23 484036 4.06
6/15/2023 14:28 483832 4.01
6/15/2023 14:33 484128 4.05
6/15/2023 14:38 483644 4.09
6/15/2023 14:44 483368 4.03
6/15/2023 14:49 484452 4.04
6/15/2023 14:54 484796 4
6/15/2023 14:59 484084 4
6/15/2023 15:04 484372 4.1
6/15/2023 15:09 483456 4.04
6/15/2023 15:14 484076 4.04
6/15/2023 15:19 483952 4.03
6/15/2023 15:24 484032 4
6/15/2023 15:29 483572 4
6/15/2023 15:34 483944 4
6/15/2023 15:39 484064 4
6/15/2023 15:45 483080 4.02
6/15/2023 15:50 483692 4
6/15/2023 15:55 483584 4.01
6/15/2023 16:00 483096 4.06
6/15/2023 16:05 482964 4.1
6/15/2023 16:10 483596 4.02
6/15/2023 16:15 482340 4.05
6/15/2023 16:20 483164 4.01
6/15/2023 16:25 482572 4.01
6/15/2023 16:30 483024 4.03
6/15/2023 16:35 481916 4.02
6/15/2023 16:41 482524 4
6/15/2023 16:46 481932 4.04
6/15/2023 16:51 482648 4.05
6/15/2023 16:56 481692 4.05
6/15/2023 17:01 481824 4.11
6/15/2023 17:06 482232 4.06
6/15/2023 17:11 483028 4.01
6/15/2023 17:16 481808 4
6/15/2023 17:21 482808 4.06
6/15/2023 17:26 482160 4.12
6/15/2023 17:31 482608 4.11
6/15/2023 17:37 482708 4.04
6/15/2023 17:42 482248 4
6/15/2023 17:47 482056 4.01
6/15/2023 17:52 482256 4
6/15/2023 17:57 481692 4
6/15/2023 18:02 481820 4
6/15/2023 18:07 481988 4
6/15/2023 18:12 482520 4
6/15/2023 18:17 480828 4.02
6/15/2023 18:22 481328 4
6/15/2023 18:27 481028 4.01
6/15/2023 18:32 481080 4.02
6/15/2023 18:38 482360 4
6/15/2023 18:43 482316 4
6/15/2023 18:48 482124 4
6/15/2023 18:53 481848 4.01
6/15/2023 18:58 481752 4.02
6/15/2023 19:03 481920 4
6/15/2023 19:08 481264 4
6/15/2023 19:13 480564 4
6/15/2023 19:18 480588 4.13
6/15/2023 19:23 480676 4.05
6/15/2023 19:28 480696 4.01
6/15/2023 19:33 480820 4
6/15/2023 19:39 480436 4.01
6/15/2023 19:44 479900 4
6/15/2023 19:49 480412 4
6/15/2023 19:54 480464 4
6/15/2023 19:59 480164 4.03
6/15/2023 20:04 479920 4.02
6/15/2023 20:09 479652 4.01
6/15/2023 20:14 478956 4.08
6/15/2023 20:19 479916 4.07
6/15/2023 20:24 479592 4.09
6/15/2023 20:29 479820 4.08
6/15/2023 20:35 479348 4.04
6/15/2023 20:40 478816 4.04
6/15/2023 20:45 479144 4.08
6/15/2023 20:50 479516 4.08
6/15/2023 20:55 479348 4.03
6/15/2023 21:00 478648 4.15
6/15/2023 21:05 478456 4.08
6/15/2023 21:10 478924 4.02
6/15/2023 21:15 479028 4.01
6/15/2023 21:20 479140 4
6/15/2023 21:25 478876 4.03
6/15/2023 21:30 479156 4.02
5

Yup, if you search the forums you'll see many mentioning a memory leak somewhere. I gave up and just reboot when things get slow. Plus I have like 3 hubs to split the load, because I love the platform, faults & all.

6

That memory consumption doesn't look bad at all. Even at the lowest point it is at 479mb of memory free. My hubs generally hover around 200mb once they stablize.

Keep in mind memory usage behaves differently between windows, linux, and Java Virtual Machines

Windows for instance we are use to seeing significant memory free, but most other OS's consume the memory much more rapidly as data is loaded and only remove data from memory as needed or as it is rolled out based on certain rules.

The hub runs in a linux based OS that is running a Java VM. That Jave Environment handles memory in a multi teir approach.

What allot of people percieve as a memory leak probably isn't.

2 Likes
7

I wonder if you were able to isolate which app is causing issues by moving apps from one hub to another? I'm thinking of buying a second hub just so that I can move the suspect to the other hub and confirm it's issue before I spend more time reimplementing open source packages.

8

Update: I disabled my "Alarm System" device and rebooted the hub. Within 13 hours, I'm back to severe load again.

Gonna keep going down the remaining list of devices and disable them one-by-one. I'll keep everyone posted.

Hub_Info_06-17-23

Date/time Free OS 5m CPU avg
6/17/2023 12:13 525196 0.09
6/17/2023 12:18 523988 0.13
6/17/2023 12:23 524188 0.06
6/17/2023 12:29 524360 0.11
6/17/2023 12:34 524156 2.02
6/17/2023 12:39 523948 3.29
6/17/2023 12:44 524308 3.77
6/17/2023 12:49 524156 3.96
6/17/2023 12:54 524080 4.01
6/17/2023 12:59 523876 4.03
6/17/2023 13:04 523688 4.03
6/17/2023 13:09 523240 4
6/17/2023 13:14 524144 4.05
6/17/2023 13:19 523924 4.08
6/17/2023 13:24 523884 4.04
6/17/2023 13:30 523596 4.03
6/17/2023 13:35 523600 4.02
6/17/2023 13:40 523848 4.03
6/17/2023 13:45 524116 4.06
6/17/2023 13:50 523308 4.12
6/17/2023 13:55 523452 4.05
6/17/2023 14:00 524036 4.05
6/17/2023 14:05 523884 4.02
6/17/2023 14:10 524040 4.04
6/17/2023 14:15 523420 4.04
9

It is very odd that your memory is so darn high and you are getting sever load. I suspect you have some bad code happening.

@support_team may need to look at you hub for race conditions. There is clearly something happening.

Do you have any rules that have cyclical logic that may get them in a loop over time. Especially if it can be triggered multiple times.

You may want to start to disable rules or custom apps to see if it helps. You could also create a backup and delete all of the custom rules/apps to see if it still occurrs. Then gradually add them back one a time keeping track of what was added when.

Be very careful of apps that use time or intervals like every 5 min as triggers as they can get nasty if they stack up. Always try to trigger off events if possibl.

1 Like
10

Nope, no rules that I can think of. Everything is based off of trigger events like: at sunset, when alarm is tripped, or a specific time of day, and I really don't have that many. Very simply setup at this point.

Apps
Apps1378Ă—604 62.5 KB

I like the idea of deleting all the custom rules/apps just to see if it still occurs, and then going from there one-by-one. I'll get a game plan together to try and test that out. If support is able to look as well, that would be super helpful.

Thanks so much for all your input so far!

11

If you click on the X in the upper right next to where you can expand or consolidate the app groups you will get a option to diable each app individually. That may help you create your plan.

12

Update: After a few days of trial and error disabling specific apps/devices, I was still seeing severe load after about 10-12 hours. Last night, I disabled ALL apps and devices in the interface, and lo and behold, after about 11 hours I'm back to severe load again.

At this point, I'm not too sure what could be the cause. Is disabling the app/devices essentially the same as deleting them completely from the hub, or would that be a next step to try? Also, while I have done a Soft Reset on the hub, I have not performed a "full reset" by resetting the Zigbee and Z-Wave radios, etc, which may be something additional to try.

Before doing a complete reset and nuking the whole setup, would someone from @support_team be able to assist with looking at the engineering logs to see if this may be a hardware issue?

13

What you may want to try is doing a soft rest and don't restore the database. I wonder if the hub would get into the bad state then.

Do you use Zwave. I think i have heard about some folks that have had sever load when a device startes to misbehave. Basically flodding the zwave network with nonsense and then causing the hub to have problems processing it.

14

Doing a Soft Reset without restore AND without resetting the radios would potentially create more problems, as all mesh devices remain connected to the respective radio, even though no devices are showing on the Devices page.

While that could be true, the best place to isolate where the severe load is coming from is to check the Logs page (Apps and Devices tab).

Don't nuke the set-up just yet, check the Logs (Apps and Devices) or post the screenshots here. Or send me your hub id and I could look at engineering logs in the morning.

1 Like
15

Yes, I do have a few Z-Wave devices --- specifically a few "First Alert ZCOMBO" Wireless Smoke and Carbon Monoxide detectors.

Thanks so much @bobbyD, I'll PM you my hub ID. Also regarding the screenshots, I've posted them below. This is after a reboot so currently the hub is not running with severe load. I can re-post new screenshots once the load shoots up if that would be helpful. From what I've posted before, if I'm looking at them correctly, it doesn't seem like the "% of total" number is high on any of the devices/apps, which is why it was not clear to me if any of them were causing the issue.

In addition, when I had all the device/apps disabled, these were showing "0s" across the board and the hub load still escalated to severe.

App_Stats_06-20-23
App_Stats_06-20-232131Ă—857 69.9 KB

Device_Stats_06-20-23
Device_Stats_06-20-232156Ă—819 61.3 KB

16

What’s the device “Alarm System”? Is that a LAN integration?

17

Yes, it's a direct LAN connection to my home alarm panel. The "Alarm System" device is using the QolSys IQ Alarm Panel Drivers

18

What i was teying to do was to isolate the devices from all the automations he has. If they were all completely removed and it still happened then we know it isn't any of them. That said i suspect the best way to get were i was thing would be to simply create the backup and the. Remove all apps.

To me this screams some kind of race condition that is getting repeated, Just dont have access to the logs to validate.

19

I know the "standard" solution to the mysterious, but reoccurring, hub slowdown is to reboot it. I just installed 2.3.5.152 the other day and that night I had the dreaded 20-30 second response time. This morning I pressed one of my Inovelli Red Dimmer switches, and I saw the LED light up to 100% but the rule that triggers didn't run. I then turned it off from the switch and turned it back on and it started running just fine. This is with a C-8, BTW. I'm disappointed they didn't increase the RAM or CPU, as I suspect this may well be the heart of the problem. In any event, didn't upgrading to .152 reboot the hub, and therefore should have prevented, not seemingly, cause this problem?

20

Unlikely as even with majorly loaded hubs don't use more than 10-15 percent of the cpu (maybe around 30 with a spike) and I've never seen anyone use all the memory. Throwing a different CPU and Memory at something is not always needed. What likely is causing the memory leak in your case is a rogue app or device. Tracking that down can be a pain. I would start with disabling any 3rd party apps you have going on, then rebooting. Wait a few hours to see how things behave then start enabling one app at a time and waiting for a day to see if there is any significant memory changes. Then enable another...wait....rinse and repeat. Monitor both logs and event logs as well for device issues. Change your backup schedule to once every 3-5 days for local and cloud.