Automated Test Suite or Stable Channel

Continuing the discussion from Release 2.3.0 Available:

May I humbly request the addition of an automated test suite around the z-wave database (or more)?

Or even a 'stable' and 'testing' branch instead that I could set as a user setting would be nice.

I believe this is at least the third breaking release in the past ~60 days, depending on how you count them. I am quite thankful the bugs are caught and fixed within a few days, but I'm at the point where I have prepared to switch to another platform entirely because the basics just keep breaking and I spend multiple days trying to figure out why my z-wave network is not working again. And I am rapidly losing stakeholder buy-in from my spouse.

I reset my entire 50+ device z-wave network in October to get rid of persistent ghost nodes from the early C7 days only to run into every broken release (i.e., 2.2.9.133 through 2.2.9.138, all 2.3.0.X until 2.3.0.119) while trying to re-add nodes, add new nodes, and stabilize everything.

2 Likes

I've made a similar request in the past, which fell on deaf ears. I'm now in the process of migrating to Home Assistant, where these small bugs seem to get caught by unit tests before release, and major breaking changes are explicitly called out in release notes.

Good luck-- stakeholder buy-in is what drove me to find a solution that was a bit more hands-off.

2 Likes

What bugs have been introduced in any of these releases that caused your Z-wave network to not work for multiple days?

FWIW, I've updated my hub to every new release over the last 6 months, usually within a few hours after it has come out. And I'm on the beta channel, so I may run a dozen beta builds before there is ever a release. Maybe I've been lucky, but I've not yet hit a bug that impeded the operation of my hub or any of the dozens of automations that it implements. Never once has my Z-wave or Zigbee meshes been adversely affected - and I've got well over 120 devices spread across the two.

I even noticed one of the bugs that caused 2.3.0.118 to come into existence, and it wasn't a big deal.

6 Likes

They do have beta test, and release. And nothing says you have to update at all. You could stick with or roll back to the one that works for you.

I am in a near identical situation. Two hubs, probably close to 100 Zwave devices, beta tester, and have never encountered any issues. So we are either incredibly lucky, or there is something else going on with your Zwave.

5 Likes

I would say that the staff does listen to suggestions. However, not every request is able to be implemented, or is just low priority for such a small team. I can't speak for them as to why they have or have not implemented your suggestion, but Hubitat is much better than most hubs in that you can choose if or when to upgrade or even downgrade, they just don't (easily at least) allow you to pick and choose a particular release to be on.

Home assistant has hundreds if not thousands of people developing (or contributing) so that might not be a fair comparison to the few person Hubitat team. And Home Assistant still manages to break things and make it hard to do upgrades when they start mucking around with their code. I don't think the grass is always greener...

4 Likes

I should add, if you are really having this bad of a Zwave issue, you probably could or should seek help here and/or Hubitat support before it gets to the point of frustration. Zwave shouldn't be bombing out for you, many of us use it successfully.

It doesn't appear that you have asked for help on this forum with your issues, there might be something the community can spot or help you with to make things run better if you post screenshots of your Zwave details page. It also might help if you post screenshots of any unusual usage in your devices or apps stats in the Logs tab, or any error messages in the logs.

1 Like

"We don't need more quality because it works on my machine" or "bugs in the core functions are fine because you don't have to upgrade" are both hard arguments to win...

Specifically, my issues usually begin from attempting to add devices to the z-wave network on a broken release where it usually will fail and/or create more ghost nodes. The Hubitat changelogs specifically show the root cause of these failures end up being issues directly related to adding nodes or database corruptions - to which I do wish to reiterate that I am very thankful are being closed quickly!

My z-wave devices always had some oddities on the C7 starting in Oct/Nov 2020 and I suspected the ghost nodes that occurred early on were the issue that kept causing instability. Issues with the C7 are now heavily documented throughout these forums. I started with about 30 z-wave devices last year and in this year in Oct 2021 I began adding another 40 z-wave devices, but I felt I had to fix the instability of the z-wave mesh first by resetting and starting over.

When I started to re-do my entire z-wave mesh, I was on 2.2.9.133 and because of the blank security confirmation page on this version and the past issues with S2 devices on the C7 I ended up switching to use SmartStart for everything as a workaround for the blank security page.

While on a release version that added nodes correctly (between 2.2.9.133 to start and then somewhere at/above 2.2.9.137), once I re-added all of the original 30 nodes everything was amazingly better - more stable, much faster to respond, etc. - with the newly formed mesh.

On the 2.3.0.115 or 2.3.0.116 release, I attempted to add 3 new devices that completely failed to join correctly and after several hours troubleshooting I finally got them to join somewhere between downgrading to 2.3.0.113 and resetting the devices multiple times. Then afterwards the rest of my z-wave network decided to go haywire and I had to air-gap about 8 different switches and repair the mesh. The issues continued over the past week with devices not responding. At one point my wife reported even several WiFi devices connected to the hub also decided to go crazy. This got to the point where one z-wave switch froze and then appears to have somehow completely dropped off the z-wave mesh. After attempting to rescue it for several hours, I ended up factory resetting it and left a ghost node that I could not clean up until the 2.3.0.119 release that dropped ~8hrs ago.

Additionally, my point is not to debate the topic of whether a stable mesh and automations continue to operate in stasis, but more to point out that the basic/core functions (e.g., adding a S2 z-wave device, etc.) should assuredly work every release and a typical user should not be worried about accepting an update with the thought that they may lose the ability to add nodes (again).

Pointing out that you've survived all of these point releases because you haven't needed to perform the action that became the bug directly fixed in the next point release doesn't mean all of those releases were Production ready / "Stable" if a core function was not working.

I am advocating for, at a minimum, a list of the core function tasks that must be confirmed to be functional on each release before it is promoted, and, as a suggestion, it would be very useful if this is implemented as automated tests in a CI/CD pipeline, because each of these versions I just named in the last 60 days should never have made it past the simple test of: "can you add a new z-wave node?". I'd be happy to track to a slower release cycle for the assurance that the core functions will never be broken on the Stable channel.

Hmm :rofl: Good luck with that.

3 Likes

Unless I’ve missed something there hasn’t been a change to the Zwave SDK on HE since they went to 7.15, so your “instability” may have other causes. Given what I perceive as a very low tolerance for disruption, I’d recommend that you consider staying at least 1 full release back, i.e. if you’re on 2.2.9.x don’t go to 2.3.0.x until 2.3.1.x is out.

7 Likes

This sounds like a good recommendation to me.

Every major release undergoes internal testing by staff, then public beta testing by users who have volunteered for that task. After a major release is let out into the wild, bugs that were not previously identified are usually patched very quickly with hotfixes that are released as bugs are reported.

Staying one release back would allow one to avoid all of that.

5 Likes

Compared to my Hubitat experience... it actually has been, now that I've gotten all my automations migrated over.

1 Like

I think @Angus_M was alluding to the notion that most people would not consider home assistant to be a hands-off solution for home automation.

To that point, here is their release announcement for last month’s platform update. Notice there is an entire section devoted to “breaking changes,” suggesting it’s not exactly an uncommon event.

4 Likes

I had a fully functional Home Assistant installation a few years ago that was irretrievably broken by a system upgrade, which made me evaluate Hubitat. I still have a Home Assistant installation, but it is ancillary to my Hubitat hubs. And Home Assistant Device Bridge makes it simple to bring devices from Home Assistant into Hubitat.

3 Likes

You can choose which version you update to. This is the key thing that is missing from Hubitat for me. How can a good faith argument be made to "just stay a major version behind", when, when you do finally update, you're forced to update to the latest release.

I very much appreciate that they explicitly call out breaking changes.

I'm not trying to get anyone else to change-- I was simply commiserating with OP and saying that I had personally found a better solution for my own use case and expectations.

A fair point. I guess a more precise suggestion would be to update to the latest version after several weeks or months have passed since the last hotfix. Most bugs are reported and fixed within days to a few weeks of a major release.

Not updating at all is still a valid option for some, since not every platform update adds something of value for every single user.

I appreciate that the HA devs call out their breaking changes too. But I appreciate more that intentional breaking changes in Hubitat platform updates are extremely rare.

2 Likes

Quite the opposite. Rule 2.5 rules still work in 2.3.0.119 ….

2 Likes

I actually have ceased being "johnny-on-the-spot" with upgrades ...almost to the point of concerned paralysis. I was in here this morning considering an upgrade because I have time right now to manage any negative consequences. That's a lousy way to look at HE ownership for the masses.

I have critical seasonal uses (like freeze protection right now) which have increased my fear of system upgrades; and with key Community Apps in the mix my fear runs both ways depending on the developer- NOT keeping up, or upgrading too soon. Catch-22

I wonder what the sentiment of HE owners is in different camps on this subject. I wonder if the majority wouldn't mind if the frequent incremental upgrades X.X.X just focused on adding devices, drivers, and material bug fixes, and X.X upgrades were when the real changes (after a lot of testing and preliminary distribution to a Developer Distribution List ) were made.

Somebody is going to step up and say, "well, that's how it is now". ... I'm not feelin that.

The other thing to keep in mind, is that unlike a lot of platforms, HE does provide an easy way to revert to a prior release (and not just the release you were on prior). So an upgrade doesn’t need to be a life long commitment.

5 Likes

Put most of the blame on Silicon Labs. Z-Wave 700 was released far before its time. Ghosts are created and only reliably removed via a secondary controller (Z-Stick). Nuking the Z-Wave database works, but is not necessary. Don't run S2 or S0 unless required (locks, etc.). Even S2 has implementation issues in some devices. The ideas behind Z-Wave are fine. But their implementation leaves a great deal to be desired. A single device falling off the network can screw up a mesh? Telling people to "run a repair"? All these issues will remain to some degree with Home Assistant. Z-Wave will have a significant marketplace issue if Thread/Matter works as promised. There's probably a reason Amazon isn't using Z-Wave for their Sidewalk technology.

One area where users have figured out a potential Hubitat specific issue is the antenna implementation. There's a great thread on adding external antennas. Many people have said the new antennas have made an appreciable positive difference in their Z-Wave performance.

4 Likes

I continue to do this with most platform upgrades and suffer essentially no consequences.

It 100% won’t be me. I understand some people have issues (and Hubitat staff work hard to try to improve on that). But that hardly means most platform updates are a train wreck for most users.

This.

2 Likes