Z-Wave storms - looks like HE Hub's fault?

@MFornander You said the 2nd screenshot is when you controlling a group device, correct? So, I would assume you are turning the group device on, correct?

What other devices do you have in that group? Are they also all z-wave plus switches? Dimmers? I wouldn't just isolate your troubleshooting to this device only but I would look at what else in that group could be the culprit. A network problem caused by one device can cause issues with another device.

I would drop devices one-by-one out of the group and see if the problem disappears, adding back in the device if the problem is still present. That would tell you definitively if the issue is being caused by another device in the group.

Is accurate to take this as meaning c5 and earlier hubs will no longer receive zigbee and zwave enhancements moving forward?

1 Like

It's my impression that the Stack is made by Silicon Labs. It 'matches' the features of the Chip, usually. The 500 series chip is quite mature. I'm not expecting huge improvements in the 500 series stack... but I'd LOVE to be wrong. I'd love to be Very Wrong, actually :smiley:

3 Likes

I'm not sure about that. If that were the case wouldn't the stack in all hubs with 500-series chips be the same?

SiLabs creates the Reference Model. Yes, anyone brave enough to fork the code probably could, but at this point in the 700 series, I'm not sure that's the right answer. Usually at this point, Hubitat would have access to SiLabs developers and could make suggestions, enter tickets, etc. just like we do.

1 Like

Yes, 500 series stack development is end of life.
The zigbee stack is the same on the c7, development there will continue as required.

4 Likes

There are 4 Zooz on off switches, 2 ZEN26 and 2 ZEN21. I'll give that a try and see what happens. I also have one of the switches mirrored off of one of the others. That might also be an issue.

Not that I think there is anything wrong with the Zooz (far from it) but there are firmware updates for these switches. Have you updated them?

Also, what firmware IS the Zooz on, and what driver are you using? With the latest Zooz firmware, the built-in drivers don't seem to work correctly, and/or give errors. Bryan Copeland's updated (community based) drivers DO work with the newer firmwares.

If you update what Zooz firmware are on, we can take a look at if there is something you might want to change.

The ZEN26s are firmware 2.3 and using the @bcopeland driver. The two ZEN21s are older and are using Firmware 20.15 and 20.16. According to the Zooz change logs, firmware updating to the ZEN21 wasn’t available until version 3, so I’m out of luck there. But after more diligent watching of the logs during a Z-Wave repair I notice an issue with the GE switch that was repeating itself and its closest physical neighbor (a ZEN21). The both hang up at "Repair setting SUC route". That shows up 4 times for each switch and then it moves on to the next one inline. I always just looked at the end of the logs where it says Finished Z-Wave Network Repair, I guess I was expecting information on any failures.
I believe that this was caused a brief experiment of using an old Wink Hub as a secondary Z-wave controller. The Wink hub was also collocated with these two switches but has been removed. I’m going to exclude and re-pair both of these switches (tomorrow) and see if it corrects anything.

I just checked - I also have a node that repeats "Repair setting SUC route" 4 times, and I think it's in a problematic location (in my garage).
Hmmmm......

That’s where mine is also.

@MFornander Surprise! Were you surprised?! Hubitat's stance is that the C5 Z-Wave stack is end of life. This problem you're finding now is because you have a non-lab, high device count, more diverse Z-Wave mesh. The problem is not new. It is very old. It has been occurring probably since the beginning.

I raised the issue multiple times. It's the reason I got a Zniffer. At one point I found 10 or 15 current threads of users with this or similar issues and posted them in a PM to some Hubitat staff. They offered to look at my logs but they would not address anything else. In the end they found nothing in my logs.

This problem is specific to Hubitat. I've proved this by taking out the C4's stick and running it in another software. The issue didn't exist at all. There is a problem in their implementation. It often creates panic in the mesh by not replying with ACK or be receiving ACK but not acting on them. I had hoped it would get better and I had told countless users with similar issues it would hopefully get better over time... but you have your answer now that it won't. You have a chance that this is solved on the C7 but I wouldn't upgrade if I were you.

I'm a little disgusted with the decision that the 500 series Z-Wave stack is end of life. If I was a Wink refugee or similar that had just bought a 500 series hub I'd be livid. No fees but who cares if the lifespan of the hub is 2 months. Retail to no support in 2 months and that window only lasted because of the brief shortage.

2 Likes

I don't think that Hubitat staff has said what that statement means. And I can interpret what Mike said a few different ways. If I read it one way, the Zwave foundation isn't developing the stack any further. Another way you could take this to mean Hubitat isn't developing it. I can even read a couple other things into what he said.

In any case Hubitat has not said they aren't doing Zwave bugfixes, or adding new Zwave devices on the C5 and older hubs. So until we get a more defined statement, I think it is premature to call the older hubs dead.

4 Likes

Correct. What has been stated clearly is that the new z-wave stack in the C7 will not be back-ported to the C5. So C5 hubs will never support S2 pairing.

And, to be fair, Hubitat Inc. never claimed that S2 support would be coming to the C5.

6 Likes

I know that there are probably (possibly) more bugs in the zwave implementation of Hubitat.
However, I take Bruce and Hubitat Inc. at their word - if these bugs can be identified, and reproduced - they will fix them. It's in their best interest.
And, they have always said that they are listening to their users. I think that they are.

8 Likes

:ear: :eyes:

12 Likes

This is not the first time the statement has been made. It has been made by Bruce and made in a way that there's really no room for misinterpretation. Maybe the stance will change but the stance currently is definitely that they are not improving the Z-Wave capabilities of the C4/C5.

If you bought a C5 hub just a month ago do not expect S2, proper beaming support (so no fix for pretty much any of the Z-Wave locks) or new 700 series devices, or SmartStart. Not to mention they will never bother to fix the deeper Z-Wave issues that have been brought up countless times in countless threads (like this one) about poor Z-Wave performance, duplicated events, mesh panic, slowdowns, trouble pairing, etc. Always... "it's a mesh issue" and no attempt explore because they won't reproduce it. They can't. I think they've all admitted to using Lutron pro bridges for most of their lighting so they just don't have the volume of Z-Wave devices to have the hops and the complexity where these issues occur.

I dunno. Nothing official or in writing but it was always eluded to as the plan, the future or at least a goal they were working towards.

Not really. The users are listening to the users. They do most of the support work for the Hubitat platform. Here in this thread compelling evidence again of a bug that has existed for years is shown and Mike again classifies it as a mesh issue. I have in at least three threads documented the issue in this thread or one that exhibits itself in a very similar way. It is the reason the "zniffing" thread exists. I had to learn another way to prove that it was their problem. It is the reason I have not one but two separate packet sniffing solutions. I can prove definitely, without argument, that the issue is the hub. How? By simply removing the C4's Z-Wave stick and performing the exact same action in PC Controller or another software that causes the issue in the hub. When it's connect to the HE hub... total panic on the mesh network. When it's not connected to the HE hub... bliss. Chuck turned on some additional debug logging the final time I proved to them it was their problem but it did not result in any further action. I can prove it on top of that with packet traces from all points of view of the conversation. But those don't even matter because I can just prove it by taking the stick out of the hub and using it in an external software. I can reproduce this bug any time I want and so can many other users.

The Hubitat staff make it your responsibility to prove there is a problem. Sure, they might listen but they are very choosy about what they do after hearing a problem.

And let me say that it's weird being on the other side of the fence from a lot of you. I consider you good people and the community is the only thing that makes this hub worth using. Your faith is misplaced and your hope is largely misguided.

  • The hub's maintenance window essentially renders it ineffective 1/24 of the time. It can't be used to monitor anything vital with a blaring, reoccurring, built-in downtime (which is totally unconfigurable when it will hit)
  • The Z-Wave issues with S2, beaming, frequently listening slaves are unbearable. This is such a crushing limitation that the community has invented all kinds of rationalizations and workarounds for why Schlage are bad locks or why you shouldn't use a Z-Wave lock directly attached to the hub. It's all nonsense.
  • The Z-Wave concurrency issues and slowdowns when you are trying to control more than one or two devices at a time are crushing limitations
  • The Z-Wave limitations and slowdowns when mixing slower Z-Wave devices (non-plus) are a crushing limitation.
  • The Zigbee issues where repeating bulbs can't be used... or dropped devices even when "known" good repeaters are used. Or the Zigbee radio just being gone sometimes after high hub utilization.
  • The poor storage performance that contribute to corruption and database issues
  • The poor performance where it is literally suggested that a user buy a second or third or fourth hub to distribute load...
  • The degradation of every aspect of the hub after sometimes just 1 or 2 days of up-time...

So, this is the short list. There's so much more. What exactly was I trying to defend all this time I spent on the forum? These are large platform issues. The solutions commonly provided by the community are often not rational workarounds. If somebody takes a step back then the picture becomes clearer. This platform is a mess.

My favorite movement that the community seems to be adopting lately for performance issues is to avoid using RM when simple automations can be used. That to me is just the icing on the cake.

I am not your enemy. I am not attacking you so don't get defensive and retaliate. This is the state of the platform if you read through the topics or own one of these devices. I'm not going to listen to anything anybody has to say in defense of these problems because I've heard all of the same excuses for more than a year. This is the way it is and I'm not alone in the way that I feel.

When I closed my repositories a couple of months ago so many people (so many people) reached out to me and expressed frustrations with the platform and with Bruce. You guys have the power to hold him and Hubitat accountable. You can make a stink about the fact that they won't be back porting functionality to the C5 hub or you can make excuses for him. If this decision stands I think it's the beginning of the end.

"Wink refugees" fleeing to Hubitat won't be paying a monthly fee anymore but they will be paying for a new hub just a month after they bought the first one to get functionality that should have been provided for Z-Wave certification long ago. I have cheese in my refrigerator that has a longer shelf-life than the Z-Wave stack of the C5.

10 Likes

Then I won't bother to point out the parts of your post that are opinion/subjective, and not universal truths for everyone.

Well, not quite. For the subjective items you only present one side of the story - your side/opinion.

For the others that will listen or want to discuss... While I agree with a large number of his points, I don't agree with them all. I use and develop on 4 different zwave and zigbee platforms, and I do believe a number of his points are true though.

But just because he says "that is the way it is" does not make all of it universally so / a given truth.

And even with some of the issues, it doesn't necessarily mean that Hubitat isn't fit for purpose/use for many people.

Anyway, I thank @codahq for all his past contributions, and I wish him good luck. I assume that now that he's gotten all his grievances out, he will be done posting here (otherwise he would just be trolling, and I think he is better than that).

8 Likes

Not to keep dragging this out... but I don’t completely follow this point. Isn’t this a risk we all take any time we buy any piece of technology and even non-tech items? You could buy a cell phone today with a new one released tomorrow. Sure it’s upsetting and you might get future software updates for some period of time, but that is not a universal truth (and I don’t think I’ve seen anyone say that C-5’s won’t be getting any updates). Same goes for cars, you could buy the latest model year and then a month later something new is announced. Sure it’s frustrating, but isn’t that just the way purchases work? You buy something at a point in time, based on its current feature set.

Same goes for a lot of enterprise software, over time versions go into maintenance release and no longer receive the same types of update they previously received. Typically even have to pay some percentage yearly as support to receive those updates. Plus depending on when you make a purchase it could just happen to fall the day before a company announces the latest and greatest.

I think that’s just the risk we all take whenever we buy anything, but more so tech based items. Doesn’t really seem like something unique to this product.

Anyway, you’re welcome to your opinions and obviously have had some negative experiences and I don’t want to detract from them. You are certainly entitled to have them. I just don’t follow the the anger and thought process on this one and wanted to share my .02.

3 Likes

I have two takeaways from this thread I started:

  1. Hubitat: Train your engineers to be a little more customer friendly.

I understand the frustration on both the user and HE engineer side. Hubitat is supporting hundreds of devices and thousands of installations. Many problems are out Hubitat's control and with a small team, it's just not economically viable to figure out.

That being said, I'm disappointed in Hubitat's responses. If you find a user that is willing to spend serious time and money debugging what seems to be HE's fault, why not treat them nicely. Suggesting I should "try a z-wave repair" after posting Zniffer logs and analysis is simply just disrespectful after the time and energy spent trying to get to the bottom of the problem. Look at how Inovelli treats their customers for inspiration.

This is an opportunity lost and it's their loss. Many users here are highly paid and trained engineers that are obviously willing to give their support. How about treating us with respect and working on problems together to make this a better platform, instead of dismissing it as a "bad mesh"? I'm still an ally but you are losing me sloly.

  1. Hubitat is not suitable for larger (50+ish Z-Wave devices) installations.

This is ok and with a $150 price tag should not be surprising. Larger installations increase the chance of strange mesh issues, routing errors, and timing stress of their hub implementation. Again this is ok but just tell me that I may run into problems at this point and that support doesn't cover this with a cheaper consumer focused hub. If you have time, please work with us larger installs and perhaps collect data to track down issues, but it's ok if you don't have the resources to do so.

Still a HE fan.

7 Likes