Hub Comes to a Crawl

[gavincampbell]

@toy4rick

Next, I read an article here where power reporting devices could be hammering the logs and causing issues. I have 8 so I went in and turned on debug logging and started to have a look

I recently discovered this with my inovelli switches. I had 30+ of them installed and their was a bug in their firmware that didn't disable power reporting. I upgraded the firmware disabled power reporting on all my devices in my home and my z-wave mesh is now sooooooo much more faster thanks to that. I think power reporting should be off by default on these devices.

I don't think that is the issue here.

This hub seems to have so little on it compared to others. 2 other things you can take a look at that I've seen mentioned in the forums,

1. Temperature, make sure that its well ventilated.
2. The power supply. Some have reported replacing the power supply with something that gives a little more juice and it has solved their issues.

[bluetunamal]

Okay, I have tried all the things recommended in this thread and even referenced other threads to try to identify why the hub is inoperable every 48 hours. None of the suggested fixes worked. I have isolated the problem for now. I removed Hubconnect App and all the Apps and Drivers Code from this Hub (Which is the Server Hub). The Hub is back to normal and operating at the speed and efficiency as it should. All my devices are still paired to the Hub, so I believe it is not a device issue.

I will now begin to add Hubconnect and the drivers back on the Hub. At least I narrowed it down for now. So my conclusion is that,

1. I may have created a problem when I installed the Hubconnect APP
2. There may be a driver that is causing the problem.
3. I installed many of the Hubconnect drivers including those that I don't need yet, so I will only install the drivers that I need.
4. Once Hubconnect connects all three hubs again, there could be a problem with a rule that causes the problem when it communicates with the server hub. If this is the problem, then I will disable that hub and see what happens.
5. My Server Hub was running hotter than the other two hubs and once I removed Hubconnect and the Apps code and drivers code, it is equal to the temperature of the other hubs.

I'm still replying to this thread until I solve the problem so others can remedy their fix, if they continue to have issues. I'm at least glad that my Hub is working fine and that it is not a faulty Hub. I also recognize that this problem is so difficult to resolve because there are no two home automations set up exactly alike.

[csteele]

1. Possible, but unlikely if you mean simply adding the code to the Hub and then selecting it to be installed.
2. Just like above, none of the drivers will cause a problem all by themselves.
3. You can add as many of the drivers as you wish, unused drivers don't get run, they just occupy some memory waiting for an event that never comes.
4. Let's focus on this
5. Some people have turned the hub over so the vent holes are UP. But the heat indicates you're CPU is running hard. Which kind of reflects back on 4.

HubConnect's job is to 'mirror events" between hubs. It doesn't interpret Events. It doesn't calculate results. Using EventSocket, the "sending" hub doesn't even use HubConnect code. EventSocket is a feature of the Platform and contains EVERY Event generated by a Hub. HubConnect is bidirectional, so it's difficult to discuss in isolation. But I'll try..

A Remote Client hub has some Events that need to be seen on the Server hub. The Server hub opens a 'permanent' connection to listen to the Remote Client's Event Socket. At that moment, every Event instantly becomes available to the Server hub. As an Event arrives on the Server via the LAN, HubConnect injects the Event into the server's event queue. That's it, HubConnect is done for that Event. (It's bidirectional, so the same thing happens the other direction... the Remote Client listens to the Server's Event Socket and injects Events into the Remote's event queue.)

What does an Event do on any hub? It causes the platform to scan through the list of subscriptions and call each. A Rule would subscribe to the Events of a device. For every Event from a specific device, the Rule gets run. The Rule might end quickly as it detects the Event is for an attribute it doesn't care about. Maybe a Battery Report comes in from a Contact Sensor, and the Rule is for the contact, not battery so the Rule ends. Alternately, if an Event comes in and that requires a large Calculation/response, then the hub would be busier than for a smaller Event. Maybe "Sunrise" event occurs and that requires going out to a weather site, and gathering a fistful of values to calculate the Sprinkler schedule for the day.. obviously that's a lot more work for the Hub than a contact sensor that turns on (or off) a single light. My point is, yes, the rules being run matter. Where they are being run matter too.

I completely agree that disabling HubConnect will restore the hub to a less busy hub, because zero events are hitting the hub. It's what the Events do that are consuming resources, not their existence.

My suggestions are.. ReInstall HubConnect and all the drivers. But you want them disabled. Do that by going into each Hub Device and clicking the Off button. You do that and you will have a chance to let HubConnect run, with zero events arriving. Your hub should remain cool and responsive.

Then, one at a time with significant waiting between, Click those hub devices back On. One of them will cause the Hub to run the large resource drain, and the hub should go unresponsive and heat up. Once again, you can click that specific Hub device to Off and verify the hub's responsiveness returns.. not necessarily immediately, but once the backlog is processed.

Then it's time to start debugging which Event is causing the problem.

[bluetunamal]

Thank you for taking the time to address my trouble spot. I appreciate your thorough explanation of the Hubconnect platform and the effects of the rules. I will definitely take your troubleshooting advice on reinstalling Hubconnect. I tend to agree with you that the problem is no.4. It's a process of elimination to find the problem and I am happy that you are helping me. I'm one of those people that don't give up easily on solving a problem and also feel a sense of accomplishment once I do. I'm not sure what I would do if I don't have conflict in my life.

[bluetunamal]

I'm still narrowing down the problem that still slows down my hub. I am adding one rule at a time to isolate the problem. I started with adding 5 separate rules first and the hub shut down again once I loaded HubConnect back on the hub. So now I disabled all of them and will begin adding those first 5 rules one at a time.

But I started thinking about my rules and devices working together. Attached is a screenshot of one of my rules. As I study it, I am noticing that many of my devices are labeled in (parenthesis). When I review my rules, there are (parenthesis) that regulate the action of the rule.

Here is my question: Would the devices that are labeled with the (parenthesis) reap havoc on my rules because the ( ) should be used only in the rules and not labeling devices? Could this cause my hub to come to a crawl? If not check my screen shot rule because it was one of the 5 rules that were running

[csteele]

@patrick

Nope, free to name them how you see fit. Good note though, try to avoid special characters and you will avoid special results...

Special characters typically are:

% $ : , . # < > / \ { } [ ] @ ' "`

We do what we can to trap for these but highly recommend you don't use them in labels and names of devices.

That's a while back and perhaps the platform is trapping really well now...

[TechMedX]

Yeah I broke a rule yesterday, and now have a persistent globe variable I cannot delete, by using ">" in the GV name. Don't do that.

[bluetunamal]

csteele,

Thanks for sticking with me on this fix. I FINALLY SOLVED the problem! Through a process of elimination it was None of the Suggested Solutions above. My Hallway Light Switch Device was the culprit. After adjusting a setting (about the time this all started) it was actually not functioning right. I never noticed it, because it would go on and off properly from the motion sensor trigger.

I set the Device back to factory settings and made sure I removed it from the ZWave network. I installed it back on the Network and the Hub now works just as fast as when I removed Hubconnect.

No Support ticket or anyone else would have been able to pinpoint this problem. I had to troubleshoot and eliminate rules, drivers, and user apps ONE by ONE until I could zero in on the issue. I just wonder if malfunctioning devices are causing so many other hub "slow downs"? Or it would just be a number of things and the only way to figure it out is to start from the beginning or where you left off, before the problem began. Anyway...thanks for all your help!!

[aaiyar]

@bluetunamal

I just wonder if malfunctioning devices are causing so many other hub "slow downs"?

As @april.brandt has pointed out multiple times, most slow downs are probably self-inflicted. I know the one I had certainly was - I force removed z-wave devices without excluding them.

[april.brandt]

@aaiyar

most slow downs are probably self-inflicted.

Oh resemble that remark. This is why I try not to run any 3rd party apps besides what is absolutely necessary. (by my own preference) The trouble is that when something is malfunctioning, it will affect everything and the symptoms are hard to distinguish. My hub crashed twice in 24 hours this weekend. Found the culprit to be a misbehaving iris plug. Only because it happened to be the only thing logging at the time. It's headed to the drawer of shame.

[april.brandt]

@bluetunamal

I set the Device back to factory settings and made sure I removed it from the ZWave network. I installed it back on the Network

Pay attention to that device. If you start to feel the pain again, that switch would definitely be suspect and if it acts up again, I would consider reaching out to the mfr and seeing if there is some warranty on it.

[csteele]

@bluetunamal

I just wonder if malfunctioning devices are causing so many other hub "slow downs"?

Along with the pair of "Building a solid Z-device Mesh" documents we have, we need a "How to isolate a malfunctioning device that is adversely affecting your Z-Device network."

[jtmpush18]

Agreed.
One way I know of to make this happen is to use the PC Controller software. This also checks for "dead" zwave nodes.
It probably also can be done by zniffering.
However, both these methods are not suitable for the vast majority of Hubitat users - they are just too technical.

Although this is a major technical challenge, it's a VERY worthwhile pursuit for the Hubitat support people.

[erktrek]

Just happened to me - a Qubino dual relay stopped responding and my main hub really slowed down. Had to force removed from HE, rebooted and ran repair - now things seem back to normal . Will be replacing with some smart switches.

[mraz.camren]

I would love a writeup on those methods

[erktrek]

So I swapped the Qubino Dual out for a Zooz Dimmer and Switch now my UI is very fast. We'll have to see how it goes though.

It's interesting the serious impact a bad device (in this case 1 parent, 2 child devices) has on the system.. UI slow, delays in Maker API, erratic behavior of neighboring devices etc... makes sense of course given the technology in play but wow. I was fortunate enough that the Qubino suddenly stopped working with my "rules" or responding to UI controls so it became kind of obvious. If it had been just a wee bit more functional I might not have discovered it for a while..

[stephen_nutt]

@jtmpush18

One way I know of to make this happen is to use the PC Controller software. This also checks for "dead" zwave nodes.

How would the PC Controller software find the problem device in @erktrek's case? What process would he run?

[erktrek]

So I have both a Z-Stick which I use for the controller software and a UZB3 which I do zniffing on.. With the Z-Stick I've found and removed ghosts etc but thats it.

With UZB3 and Zniffer software I've detected slow network performance but the issue for me is trying to trace things down to the "bad" device. That's where I noticed my MS6's were slowing things down although now I wonder if the Qubino had something to do with it.

I am totally interested in this stuff and willing to play around and learn but someone else who just wants it to "work" might not be... this is the issue. How to make it easier for the average user I guess..

[ritchierich]

@erktrek

That's where I noticed my MS6's were slowing things down although now I wonder if the Qubino had something to do with it.

I’ve got both too and you may recall me mentioning my MS6s are operating at 9.6, super slow. Aeotec has never come back with any more details, I need to follow up again. I have 3 Qubinos, 2 Flush 1Ds, and a dual relay which have all been fine. At least I think they have

[erktrek]

yeah thats right.. mine were too so probably not related. Bummer that no one got back to you. The Qubino just stopped working in HE all of a sudden - physical switching was fine. The impact to my Hub was severe. I dunno how they will ever be able to isolate something like that given the nature of Z-Wave but it's definitely a real performance killer.