In the past couple of weeks I've had two incidents of "Severe Load" and one incident of "Elevated Load." I think the two big changes in that time is adding an RSTP camera and installing/using mDash. Don't know if it's related at all. In general though what I'm asking is the approach to tracking down this issue. It would be nice to have access to the "top" command in the underlying OS, but failing that...
What has anyone else done to troubleshoot and fix this issue?
Thanks!
Jay
If you enabled RTSP on dashboards that will certainly cause the CPU to go up and generate those alarms. What are the resolution of the streams you are using to for the RTSP Streams and how many of them are you running on the dashboard when you are watching it. try to run them with the lowest resolution your camera can put out and see if that helps.
I agree the tools are somewhat lacking to understand cpu load. There is allot that the current tools just don't capture.That said what I do is use HubInfo driver from @thebearmay, my Live logger influxdb syslog driver to export the logs to Influxdb, Influxdb, and grafana.
It isn't perfect but it doe give you a clue when it is occuring and a better chance of seeing something happen
In Grafana i use a Dasbhoard that @jtp10181 created and it this it the lower part of it. This information can be super important to understand the trends of how your hub is going. This is my last 48 hours. All looks good and you can see the reboots for new betas with momentary spikes lof the CPU load
Here are the graphs when i was testing the RTSP streams in beta. You can see the Temps and then CPU load very high in the beginning of these graphs. That was when i had all of my camears connected with High resolution streams. It actually caused High load and a triggered a reboot from it getting to hot. The second half of this graph was using low res streams on the same cameras i could adjust.
I would not put RTSP streams directly on a dashboard that is always up and accessed by a tablet or something like that which is constantly streaming.
Yeah, I strongly suspect the RSTP camera. It's only one camera and only in mDash. I'm avoiding viewing the camera until I figure out how to stream a lower res version.
If you want to confirm w/o grafana… Logs > App Stats and Logs > Device Stats. It’s the closest thing you’ll get to top and will show you who your heaviest hitters are in real timeish.
Are you using the ez dash or the legacy dash link? The legacy dash link is mjpeg so the hub needs to convert from Rtsp to mjpag. This consumes a lot of cpu especially if you use an image tile to display it on your home dash constantly. I also had a similar problem. My solution was to move the conversion out of hubitat and on to node red and present a url with mjpeg back to the hubitat dashboard tile, my cpu load went from averaging 30% down to averaging 5%
I also cleaned up a bunch of noisy devices and apps as I was doing that, which has further reduced the load on my hub and my DB growth.
cheers
Duncan
No, only place I get to it is via mDash.
