Hub process slowdown after several days

Packet cap proof:

asa1# sh cap jpcap

1029 packets captured

   1: 20:47:58.022948       802.1Q vlan#1 P0 192.168.7.68.55927 > 18.223.183.64.8883: P 3173872143:3173872174(31) ack 2022206854 win 312 <nop,nop,timestamp 1053925432 3355598586> 
.
.
.
1029: 21:44:57.437492       802.1Q vlan#1 P0 192.168.7.68.55927 > 18.223.183.64.8883: . ack 2022210484 win 312 <nop,nop,timestamp 1054267373 3356460924> 
1029 packets shown

56 minutes, 1029 packets = 3.2 seconds per packet.

That's one packet every 9 seconds per hub (I have 3.) Meaning it's possible to have nearly zero cloud traffic on a reasonably complex system.

1 Like

My little C4 hasn't had any slowdowns that i've noticed (I have motion lighting - but never wait over a second for it to fire). I've never restarted my hub due to slowness, only ever to try and fix something i've probably done wrong.

3 Likes

I have a simple test Rule that I can tell when things slow down.

testsw

The 2 test switches are virtual switches configured with a 1 second turn off time. I then have them on a dashboard.

When either of the switches is pushed the rule triggers at some point. Then the if statement looks to see if the switch is on and either turns the light on or off. If the light doesn't turn on or off then the switch has turned off before the if statement executes.

After a reboot the rule always works flawlessly. I can watch the log and see that everything happens within about 200 msec. But over time that time gets longer to the point by the time the rule executes the switch is already off. I point out that the rule always gets triggered but sometimes it is far enuf past the time the switch is turned on that by the time the if statement executes the switch is off.

This has been very consistent. I need to start tracking how long or how many days it takes for the slowdown to affect it.

I was hopeful as this is my 4th day since a reboot, and I have never made it past 2 days ever since I first set up HE, but this morning one of my zigbee switches was changing faster than the rule was evaluating it, which resulted in an endless cycle of my kitchen lights turning on and off. I was able to pause the rule, and it took longer than usual to access the web ui. I waited a couple minutes and checked other automations, which are usually immediate, but now were taking several seconds to initialize, so I rebooted the hub and all is well again. I have definitely made progress, but obviously there was more than one issue at play here.

Sounds like a good use case for private boolean to keep the rule from being re-run if its already running.

@csteele / somebody:
Can you please make an app/device/something which would measure the "degree of slowdown"? It would be helpful to move to an objective measure, rather than subjective feelings on this issue.

So far no slow down after changing from the Hue kitchen group to an HE kitchen-lights group in my RM4 kitchen lights rule. But it's only been less than 1 day, so I wouldn't expect a slow down yet anyway. To be honest, I'm finding that the HE group sometimes popcorns. I'm willing to put up with it for a few days in the interest of experimentation, but long term I'd rather go back to the Hue group and reboot every night.

But... I had another thought...

I've been looking hard at my logs, and I noticed that the Kitchen Lights rule is running multiple times within a very short time frame -- less than 1 second. It's fast enough that the private boolean doesn't always have the intended effect.

I know it's happening because I have three motion sensors triggering the kitchen lights. When I walk through the area, like in the morning when I'm headed for coffee, I trigger all three of them almost simultaneously. I'm wondering if the poor rule is tying itself in knots? This might explain both the slow down and the problem with one or more of the lights not dimming or turning on correctly.

Since I'm already using private boolean in that rule, I've added a local variable. Like this:

  IF alreadyRunning = True THEN
    exit rule
  END-IF
  Set alreadyRunning = True

  --- do rule stuff here ---

  Set alreadyRunning = False

Don't know if this will help -- I'm open to other suggestions -- but it does seem to stop it from running multiple times very fast. If I go a day or two more with no slow down, I think I'll change back to the Hue group and see what happens.

If all 3 motion sensors see you anyway, just use 1 instead of 3.... Then you wouldn't have to worry about concurrent running issues.

They actually cover different areas. One points toward the entry and triggers lights when I come home. One points towards the family room adjoining and open to the kitchen. I spend a lot of time there and the kitchen lights illuminate it. One is in the kitchen itself, to keep the lights on while I'm in the kitchen.

It just happens that when I walk from one side of the house to the other, I trigger all three of them. And, in the afternoon when the slow down is most visible, I tend to walk around a lot.

If they happen very close to another you will always have redundant/concurrent rule running issues. No way around that IF they happen faster than the execution speed of the RM rule... If they are further apart (a few hundred ms maybe?) You are probably fine.

It was a switch triggering motion lighting on/off. It became disoriented apparently since it has never happened before, although I’ve never had the hub slowdown happen this gradually before either.

I use the built in Zone motion controller to create one aggregated virtual motion device that I use in the Motion Lighting app for my garage lights.

2 Likes

That's an interesting thought that I think I'll take a look at. I also have multiple motion sensors in my bedroom/master bath that I'd like to explore that with.

It has been more reliable for me without hammering the logs with the motion sensors. I started using it because the lights kept turning off when I was working in the garage.

That's one of my uses. Lights stay on whether i'm in the shower. on the toilet, or at the sink. Single triggering device, single simple rule. Win!

2 Likes

Did not know that it existed. Ya learn something every day! Thx @Ken_Fraleigh and @bjcowles.

Now... once more unto the breach...

1 Like

Forgot about this. Perfect for some of my needs.
Nice catch. :smile:

You know this is my thoughts on it. I like hubitat owned it a few months now. To always blame it on 3rd party plugins I think is looking at it the wrong way. We all buy these hubs to run our automatons the way we want too. If the hub turns to crap for any reason it's not meeting the consumers needs. I really loved hubitat when I first got it then came the slow automatons the slow response time of the UI 30 - 60 seconds per click. I have my hub rebooting every few days which has helped alleviate some of these issues and caused other issues with my mesh so as far as I'm concerned it's not meeting my needs. I had better luck using ST cloud. I contacted support they helped make changes and blamed it on the 3rd party apps/drivers and closed the ticket. If Hubitat knows we are suffering why is no one officially addressing this issue? Local processing does no good if it takes longer than cloud processing due to these slow downs. I'm honestly ready to go back to ST I never had to contact there tech support for issues. Don't get me wrong they had outages but I was aware of them and aware they was working on them. It seems the only answer we get here is contact support and after a reboot it all runs good for a few days and the ticket is closed. I don't know about you but I'm not about to waste my time in this loop each time. Someone needs to officially address there is an issue if they have not already.

5 Likes

We have and continue to address "this issue". The issue is that people install bad custom apps and drivers on their hubs, and then expect us to fix that problem for them.

It's not possible for us to address the problems in user installed custom code. We have provided you the tools to diagnose which of these is the source of your problem. What you do about it is up to you. We've provided the power in this hub to do anything you want with it. That doesn't mean that the choices you make are good ones and will have good results. Most users, including those with very large systems, have no "hub slowdowns" at all. What's the difference between those with and those without? The difference is in the choice of custom things that you've put on your hub. Do you know those apps are solid? If you don't, then why would you expect us to protect you from yourself? We aren't the custom app police here.

We aren't going to devote support resources to people who won't help themselves by making good choices for what they do with their hub. We bend over backwards to help people who want the help, and who follow our advice about how to resolve their problem. But many who complain simply refuse to listen to our advice, and then go on to complain more. There's no helping some people.

Our advice is always the same: Selectively disable custom apps to see if the problems are resolved without that app running. By this means, it is possible to find the culprit, or culprits, responsible for your hub slowing down. If a person were to disable ALL custom apps and drivers, and still have a demonstrable issue, of course we would dig into it. However, in our experience, all of these hub slowdowns are coming from custom apps and drivers that haven't been designed properly. Look at @csteele's post above, where he discusses this in depth.

I should add that it is possible to design rules that get into infinite loop types of problems, and these too could cause a hub slowdown.

5 Likes

It is what it is, I guess.

That is exactly why I don't think Hubitat will ever get market share outside of the enthusiast community.

That is not a stance that will ever allow for wider adoption with consumers. Telling a customer - "well it works fine as long as you don't install anything on it" simply isn't going to work for an average consumer. Yes, I'm being a little facetious - that isn't exactly what Hubitat has said, but it is how it feels on the customer's side.

For the record, the last app I took the time to prove was slowing down my hub was Hubitat's Chromecast integration. Oh yeah, that's BETA though - so also not Hubitat's problem either I guess.

Anyway. I understand the company's stance, but I don't agree with it. I know - tough luck, buttercup.

But that is exactly why I won't recommend Hubitat to casual users over SmartThings, and why I'm migrating logic off of Hubitat as fast as possible and turning it into a zwave/zigbee/eventsocket/MakerAPI connector hub and nothing else as much as I can.

5 Likes