This may be unrelated to your issues, but have you by chance had a FLiRS device (battery powered Z-Wave locks, thermostats, etc.), that might have very low/dead battery recently?
I had a severe hub problem similar to yours during the recent beta which turned out to be caused by a battery powered Z-Wave thermostat when the batteries died and the device fell off the mesh. Once the device was offline, havoc ensued, as trying to communicate w/an offline FLiRS device is not good for your mesh:
This is a LSS (FLiRS) device .. So sending a command to it while not powered would cause a flurry of activity while it tries to wake it through the previous route and multiple other routes after it doesn’t reach it through the expected route..
Doesn't sound like your case, but wanted to ask just in case, if there were any coincident/recent battery issues w/any of your battery powered Z-Wave devices.
Thanks for the suggestion. Remind me to avoid FLiRS devices!
I run a battery report and a device inactivity report every day... nothing below 30% and no inactive devices. My locks are all zigbee and my thermostat is ecobee, so no FLiRS devices there and none that I am aware of elsewhere.
I had a similar problem but with older firmware. If a key device dies or goes. offline it can cause mesh to crash. I had a switch die that had like 14 things routing through it. Every device started panicking.and the mesh crashed. Do you have a location rule to reboot on.mesh/zwave failure?
I have a rule that will alert if the zigbee or zwave radios go offline. Neither one of them triggered. In this case a simple reboot did not resolve the issue. What did was turning the power off for 30 seconds.
I have lots of repeaters and most of them repeat only one or two devices. The one exception to that is an Aeotec zwave siren of all things. It has 7 devices routing through it, but that would not explain the 40 or more that were offline. Since the siren is mains powered I send a command to it once a day to make sure it's still there. It doesn't show any signs of having failed. While I did not do a careful analysis of the failed devices (since I was sitting in the dark, my zwave motion sensors nonresponsive) it seems to me that the issue affected both directly connected and routed devices...
My zwave mesh is pretty solid, and in better shape than ever since the 7.17.1. update. No ghosts, Most of my devices are at 100kbps, the rest at 40 save one outlier at 9.6 - a Leviton zwave switch that somehow escaped the Great Zwave Switch Migration of '20 (all of my other zwave switches were replaced with Lutron Caseta). I should probably nuke that one too.
However... your question did prompt me to remember that I removed a couple zwave devices yesterday morning - two leviton zwave plus outlets that never did work all that well - long RTTS. I excluded them and replaced them with zigbee outlets instead. I did check to see if they were repeating for anyone and they were not, or so I recall. But that is one additional change I made to the zwave network.
well... thanks to @tony.fleisher's zwave app it turns out those ring security keypads ARE FLiRS devices. One of them was definitely on the failed devices list this morning since that's how I noticed the issue... couldn't disarm the alarm system from the bedroom keypad. But it came right back when the mesh came back. The battery was just charged the other day and it's at 100% with its twin on the 1st floor at 57%.
Good secondary sleuthing on your part. Definitely keep an eye on those buggers. Seems like if it wasn't showing as failed before/at the time of the mesh going south, may not have been involved...
I wanted to emphasize this post by @rfg81 again... in case others run into the same troubles I've had.. (Many Thanks!!!)
I recently took delivery of 2 uzb7 sticks and was trying to upgrade them. Like my previous attempts was finding it impossible update the firmware. PCS OTW failed, TeraTerm failed etc..
Finally found success by rolling back the driver to the prior version by doing this in Windows 11:
First go into the device manager, find the appropriate device, call up the property page and click on "Update Driver":
Once this is done run the PCS, select the proper com port and do the OTW per usual... Not saying it will work for everyone but was able to update both sticks to .18 easy peasy.
Note: Making this change is only temporary - if you unplug the stick and plug it back in it will revert to the most recent driver and you will have to set it again. Found this out the hard way.
Definitely keep an eye on those buggers. Seems like if it wasn't showing as failed before/at the time of the mesh going south, may not have been involved...
