[2.3.5.113] C8 Z-Wave dropout at random times of the day

jakokln · March 19, 2023, 5:23am

I know there are some Z-Wave and Zigbee threads already, but I don't think I've seen this exact issue yet so I wanted to start my own.

So I am getting a weird issue with my C8 hub with the Z-Wave radio. It seems randomly at times maybe 1-2 times a day it will just drop all Z-Wave devices and they all stop working. Just like the radio was turned off. Zigbee (as well as everything else hub related) has been solid even when Z-Wave stops working.

The two methods of fix I have observed are Z-Wave repair or a power down. It seems the reboot doesn't help for some reason. I have also noticed when it fails the Z-wave topology map goes entirely red, and the repair shows tons of failed nodes during the repair process. It seems to begin working at some point after the repair is complete. When powered down and rebooted, it seems things begin working right away after boot.

I initially setup the hub on 3/3, using the most recent firmware at the time. Not sure which one it was exactly. It seemed the failures happened daily at least, and after a few days I finally got frustrated enough and plugged the C7 back in. Used the C7 just fine for about a week, and decided to play with the C8 again since I had some more time.

When I got thinks back up and running earlier today (around 1-2pm) this was my process:
C7 is on 2.3.5.113 as of a couple days ago
Cloud (migration) backup for the C7, I do not have hub protect
Powered down C7 & unplugged
Plugged C8 back in, and performed soft reset from the diagnostic tool
Updated C8 to the 2.3.5.113 firmware
Cloud backup from the C7 to the C8

It came back online and after a little while of it getting its footing it was pretty much fine. Z-Wave and Zigbee were both responsive from the dashboard and webpage. Then around 8-9pm I noticed things were no longer working and I was seeing issues in the Z-Wave details page like I described above. Ended up doing a Z-Wave repair, and it looks like the devices I tested were fine again.

Not really sure what's going on here, if it is software related or if the radio in my hub is messed up and randomly resetting for some reason. I saw mentions of a bad antenna, but I don't think that is the case because when it is working, it works really well.

rlithgow1 · March 19, 2023, 1:11pm

That's because a reboot doesn't clear the radio but power cycling for a minute or 2 will.

Can you post your z-wave details page?

jakokln · March 20, 2023, 12:17am

Good to know. Here's my details page and topology map today since things have been working. I didn't think to take screenshots yesterday when everything was acting up.

There are a couple devices I have unplugged (Xmas lights and such)

danabw · March 20, 2023, 12:41am

That is a lot of devices to lack routing when you migrated to your C8 on 3/3, over two weeks ago. Some of them look like they may be ghosts, possibly brought over from your previous hub. The ones w/the "Discover" button in particular. But really, I would have expected all of your normally connected Z-Wave devices to have routing by now, and many do not.

This is usually not too good for a Z-Wave mesh...are the devices you've unplugged Z-Wave plugs, or ? You should, especially while trouble shooting Z-Wave issues, have them plugged in if you did not exclude them from your hub when you removed them.

Plug in any mains powered Z-Wave stuff you removed, and then do the shutdown dance, and then see if anything changes in your Z-Wave Details screen.

Shut down hub from Settings menu
Pull plug (from wall, hub connector can be fragile)
Wait one minute
Plug in hub and let it boot up
Go to Z-Wave Details page and post again if it's different.

jakokln · March 20, 2023, 1:41am

Just to clarify, I did run another cloud backup yesterday 3/18, before swapping back to the C8, and that was the backup I selected when setting up the C8.

I have compared the list of devices here against my spreadsheet and yes the 3 items with the discover button are not anything else. Everything that shows here is what I had logged as active devices. What would be the best method to getting rid of the ghost devices?

I am aware unplugging the devices is not good for the mesh. Really for the Xmas lights there is just one plug, because its outdoors and a plug rated as such. This one is on the opposite side of the house from the hub, and has been unplugged for the last couple months at least.

There are two other devices unplugged, and have been for a significant amount of time (probably since 2022) They are: Old water shutoff device, and SS002. One is a Dome water valve which was no longer strong enough to close my water valve, so it got put in a drawer. The second is a spare switch I keep around incase I need to swap one out or test something quickly.

Oddly enough, a couple devices that didn't show routing information seem to be working fine. I toggled the devices on and off and now the routing information is showing.

Things were pretty weird this morning, but upon arriving this afternoon everything has worked just fine.

I will try your steps listed. Might not be able until tomorrow, unfortunately

DustyFromBMore · March 20, 2023, 2:27am

Most plugged in ZWave devices act as repeaters, so if they're missing, devices may be missing "hops"/nodes in the mesh. If you only use them for parts of the year, best to do the exclusion dance before taking them out completely (a pain, but if only done during the holidays, shouldn't be too painful). A repair after exclusion will heal any "holes" left by those devices, ideally, after about 4-24 hours depending on how chatty your devices are.

I've noticed that my new C-8 has shorter routes for some devices, which means they're getting more direct connections to the hub.

danabw · March 20, 2023, 2:34am

Not having routes for up to 3 days is normal after a migration, so if you've just migrated again I would only initially be concerned about the devices w/"Discover" buttons...

jakokln · March 20, 2023, 3:12am

I get the mesh needing time to rebuild, but is it normal for every device to just stop working on the Z-Wave network? This is what really confuses me, and makes me think there may be an issue with the radio in this hub.

So my wife just let me know she tried turning off the deck lights, and they wouldn't go off. I logged into the webpage, tried turning them off, and nothing is happening. The device is directly communicating to the hub, there is no hopping through devices

jakokln · March 20, 2023, 3:23am

Topology view is still showing normally though. Can confirm though, no other Z-Wave device is working at all, its simply like I powered off the hub.

Running a Z-Wave repair and every device is showing failed. Whether its communicating directly with the hub or going through other repeaters. I definitely know it isn't a repeater issue. I think something is failing with the radio in my hub

jakokln · March 20, 2023, 3:41am

A shutdown got everything working again, and now everything is fine and responsive. I'll keep the C8 up for another day or so, but if it keeps acting up its going back in the box and getting returned. Maybe I'm too impatient, but I'd rather have my reliable C7 over an intermittent C8

jtp10181 · March 20, 2023, 3:45am

Yes. You are.

Also, all the devices with the "Discover" button are failed pairing, or devices you force removed from the device page, but did not exclude from the zwave network. You should try to remove as many of those as possible by doing a refresh wait, look for remove button, then remove.

Judging by the fact all the "Discover" devices say 1 neighbors, they are not alive on the network. So probably are powered off and removed or got paired to the mesh as another node.

Once you do that, check for the routes, you should have a majority showing routes. Some battery devices may take days though due to the way they work. If you want to try to speed it up, you could do a fresh power down reboot and then run a repair. Just be warned a full repair can disrupt the mesh temporarily.

rlithgow1 · March 20, 2023, 10:28am

As @jtp10181 you really need to take care of those. Exclude them or remove them from the hubitat. You will not have a healthy mesh until then.

jakokln · March 20, 2023, 3:46pm

Ghost devices have been removed. Everything but one device is powered up and showing routes (after toggling them on and off routes showed up). The one device is powered down and not in use, so I will get around to removing it eventually.

This morning devices were on which should have been turned off at some point in the night / early morning. I removed the ghosts this morning, and power cycled the hub.

So far things have been fine, so we shall wait and see if everything drops again.

Thanks for the help everyone, hopefully the network will remain stable and I can enjoy the benefits of the C8 hub!

danabw · March 20, 2023, 3:51pm

Things should settle down, and even improve over time now that things are cleaned up.

jakokln · March 21, 2023, 12:56am

Well it happened again. Got home around 5:30pm from running some errands and all my Z-Wave devices were non-responsive again. Everything was totally fine from what I noticed before I had left an hour prior. Wife mentioned something wasn't working as intended while I was gone as well.

Z-Wave repair got things back up and running again, without the need to power cycle. Is it a bad idea to continually run the repair or power cycle like I have been doing? Is doing one better than the other? Should I just let it sit once its down and see if it comes back by itself?

Some routes were gone when I checked a little bit ago (7:30pm about 2 hours later). Toggling the devices missing routes again made the routes reappear. Reposting my Z-Wave details page to see if any other information can be gathered from it. The old water shutoff device is the unplugged device (towards the end of the list).

Running repair after the failure around 5:45pm:

jtp10181 · March 21, 2023, 1:07am

Yes, bad to keep running repair. Best way is to do a hub shut down, pull power for 10 seconds then boot back up.

How long was hub running when this screenshot is from? Front door lock 0x30 has a lot of route changes.

jakokln · March 21, 2023, 1:45am

Good to know, if another failure happens I will shutdown and power cycle.

The Z-Wave repair photo was taken at 5:42pm. The hub had been up and running since around 6:30am. The front door lock is a Schlage Z-Wave lock and I know I have heard people complaining about the radios in them, and sometimes needing a repeater nearby for it to work well. There is a switch in the same room, as well as 3 in the next that are Z-Wave. Though it is fairly close to the hub, about 20ft directly. I believe from my past checking, it has usually ran directly with the C7 hub.

Been fine all day otherwise, though I have been running the washing machine throughout the day, and that device is fairly chatty (power monitoring). Is it possible the increased communication kept the network running well?

danabw · March 21, 2023, 1:55am

Chatty devices doing a lot of power reporting are usually associated w/meshes that get overloaded and don't work well...but if you've been running those power reporting devices for a while previously w/out issues probably not enough to take things down (in normal situations).

A lot of route changes isn't a good sign, generally.

staze · March 21, 2023, 1:56am

I’ve been seeing similar. No ghost nodes. But includes don’t seem to work reliably and mesh seems… unstable… since migration. Last couple days have been better. Seems like if you can make it through one of the nightly “cleanup” tasks the hub does things improve. But still hoping for an update soon as things seem far too… fragile… atm.

jakokln · March 21, 2023, 3:36am

I say chatty, but its probably not too bad. I have the power limit changes set fairly high, so it shouldn't update too much. It will probably change when the washer changes modes, I'm guessing on average one power update a minute for the 45 minutes the machine runs. Usually don't see issues with it while running normally