Slow Hub - better tools to diagnose slow running devices/apps

mhutchy · June 4, 2020, 7:57am

Its soo difficult. There is no way of knowing if we are overoading our hub or not? Is the CPU at near 100% in which case off load a bit. Is the memory near capacity? is network (zigbee, ethernet, z-wave) near 100%? I understand that some people are buying 2nd hubs but I am hesitant to convince my self "Hey, the product that worked well for months is now less reliable, the solution must be to invest more in the same product?"

I have been moving rules out of RM, using simple automation and also using the "Button controller app". More complicated homeMade apps, I have moved to a raspberry pi running python.

I have a lot of tasmotta devices on my system and am thinking that the ethernet maybe a pinch point (the web interface on the hub can slow down and rebooting resolves the issue). I am going to write a tasmotta driver using MQTT as this would limit the hubs ethernet traffic to only 1 IP address which is on the same LAN hub as the hubitat. It is going to take a lot of work and may yeild no change whatsoever (as I don't know if the LAN port of my hub is at capacity or not)

I'll feedback when I have some results

Ken_Fraleigh · June 4, 2020, 1:04pm

I was rebooting daily as well 7 months ago. I decided to add another hub and put all of my 70+ zigbee end devices as well as most automations on the new hub to control my zigbee lights and z-wave plus devices via HubConnect. The new hub has never experienced a slow down, but the original C-5 hub gets rebooted weekly or it slows down. Rebooting daily is terrible; not only from a reliability standpoint, but also because the mesh never seemed to have time to stabilize and response times were slower.
I just use RM to reboot the hub at 4:00am Wednesday and Sunday and I haven’t seen a slowdown (except when I disabled this rule, which is how I know it takes a week) since adding the new hub.

dman2306 · June 4, 2020, 1:18pm

What's the emerging technology? Smart home tech goes back at least to X10 which started in the 70s and both Z-Wave and Zigbee were introduced in the 90s. Though that still seems like yesterday to me, that's 20+ years ago!

If the issue is software (which is pretty much what the HE team has said so far), throwing hardware at it is the worst thing you can do. It just masks the problem instead of resolving it.

Angus_M · June 4, 2020, 1:23pm

Yeah it seems mostly software related. Although the big players also want to introduce a new device standard/protocol and are still iterating their devices quite rapidly. And it's clear from the numbers of start ups and shutdowns in the industry that HA is definitely still an emerging, immature technology space.

jeubanks · June 4, 2020, 1:48pm

... the DIY space is where these problems exist.

mhutchy · June 4, 2020, 5:13pm

So,

I have re-written my tasmota driver to use a MQTT server, so far so good. Switches are lightning fast and I now don't need to run an automated refresh to know the switch status as it parses via MQTT. The hub now only talks to one IP address (the raspberry running the MQTT server) which should lift some load off the hub's LAN card

Increased the CPU load on my raspberry from 5% to 7% so I can not see that being the entire solution.

jpoeppelman1 · June 4, 2020, 5:26pm

Are you using web sockets when connecting to HE? Or are using a client on the HE hub?

mhutchy · June 4, 2020, 5:38pm

So my original tasmta driver used sendHubCommand to send an HTTP get to the tasmota device's web server. so a "192.xxx.xxx.xxx/cmnd=power%20on" to switch on or cmnd=power to get switch status etc.

I have now setup an MQTT server on my raspberry. And modified my tasmota driver use that server:

the intialise() routine of the driver sends the MQTT config commands via HTTP to the tasmota to setup the MQTT server
I have then used the MQTT commands (detailed here MQTT Interface - Hubitat Documentation) to send commands to the tasmota and parse the replies

This works really well, I get an instant parse with the new switch value when a tasmota device changes state (even if done locally) as I am not relying on a "runEvery... ()" to fetch the switch status

Driver code below (if your interested)

github.com

mhutchy/Hubitat-Drivers/blob/master/Sonoff Tasmotta (with MQTT)

/**
 *  Sonoff Basic
 *
 *  Copyright 2019 Mark Hutchings
 *
 *  V2 - disable driver option added
 *
 *  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
 *  in compliance with the License. You may obtain a copy of the License at:
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
 *  on an "AS IS"BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License
 *  for the specific language governing permissions and limitations under the License.
 *
 */
metadata {
	definition (name: "Tasmotta (with MQTT)", namespace: "mhutchy", author: "Mark Hutchings") {
		capability "Actuator"

This file has been truncated. show original

jpoeppelman1 · June 4, 2020, 7:11pm

Nice, can you send the raspberry pi code as well? Is your MQTT server written in node.js?

jpoeppelman1 · June 4, 2020, 7:14pm

Nevermind, I want to do something different. I'm striving to use the HE streaming event socket when connected to a node.js server . Likewise, I want to be able to send commands back into the specific HE hub using the node.js server. It's kind of like HubConnect, but without all the overhead...

csteele · June 4, 2020, 7:17pm

Why not just create a "Remote Client" app that speaks the protocol.. like dan.t did with Homebridge? The HubConnect API is documented... for exactly this purpose. At least that way all the Select Devices portion is done for you.

Additionally, there is a HubConnect proxy for NodeJS and it has a built in WebSocket.

mhutchy · June 4, 2020, 7:17pm

Hi,

I used the "off the shelf mosquito server": Introduction to IoT: Build an MQTT Server Using Raspberry Pi

Its Really easy to setup and then it sits in the background doing the work for you. The additional benefit is my python scripts can now get the info from MQTT as well therefore avoiding 2 devices (hubitat and python) poling the same tasmota device.

jpoeppelman1 · June 4, 2020, 7:31pm

I believe I'm looking for a more generic solution. Using web sockets we have a way to extract all events to a node server while caching state.. Sending commands from the node server back to HE via web sockets is where I'm struggling... Maybe that is where things break down meaning there is no way to send a device event generically from a HE smart app. Meaning, the devices within the HE app need to be inputs... I realize maker will allow http commands, but I want sockets!

rsjaffe · June 4, 2020, 8:00pm

Something like the linux top command would be nice to get an initial idea of what's consuming hub resources when it's slow.

csteele · June 4, 2020, 8:04pm

Useless.. sorry

All it would tell you is what you already know.. that Java is running, Java is using all the resources.

aaiyar · June 4, 2020, 8:04pm

The tool would have to something that works within the JavaVM that the platform runs on. top or htop wouldn't be informative in this instance.

jeubanks · June 4, 2020, 8:08pm

Oh OH I know lets run Dynatrace within the hub and then the license alone would cost more than the hub!!!!! Everyone would get a MUCH more expensive hub with ability to see resource usage!

Reality is that running a trace attached to the JVM would require even MORE resources bogging down the hubs even more.

aaiyar · June 4, 2020, 8:20pm

jmap output could be useful, but not easily interpretable by the end user ....

jpoeppelman1 · June 5, 2020, 11:04am

After reading through many hub slow down threads, I have decided to stick with HubConnect and move all RM rules to a dedicated hub for processing.

Question - in order to achieve the lowest latency possible and reduce the probabiilty of a hub slow down, should I move all zigbee devices including my Hue hub lights to their own dedicated hub and then disable each respective radio? 80% of my devices are zwave...

My current setup:

(8) Hubs (yes, 8 hubs, I'm sure Hubitat Inc. loves it!)

HubConnect (some RM Rules)
1st Floor (zigbee and zwave devices with a splash of RM)
2nd Floor (zigbee and zwave devices with a splash of RM)
Custom-Apps (very risky apps with high poll frequency)
Media-Apps (Sonos, Harmony Hubs)
Cloud-Apps (SharpTools, Google Assistant)
Alexa-1 (dedicated hub for controlling TV channels in specific room on separate Amazon account, Alexa turn on ESPN...)
Alexa-2 (same as above, only for Master Bedroom)

Angus_M · June 6, 2020, 11:56am

My God. Eight hubs and you are still worried about performance?!? I'm still debating adding just one more due to the occasional seizures and slowdowns!