I have five C7 hubs in my home. All are running Platform Version 2.4.3.173. All have a similar suite of devices connected. All are about the same age. However, about one week ago, one of my C7s began experiencing this issue: All devices with a physical switch stopped reporting an accurate Current State when the state is changed via the physical switch. This is consistent across several device types.
All of the digital calls work as expected and those statuses are accurately captured. All of the sensors are working as expected. All of the relays (no physical switch) are working as expected.
When I go to Devices -> %Any Z-Wave Switch% -> Commands and click "Refresh", the accurate Current State is retrieved.
No new devices were added or removed immediately preceding the onset of the issue (~10 days)
No Platform update was done immediately preceding the onset of the issue (~10 days)
I've tried:
Changing to/from Generic Drivers
Air gapped the switches
Power cycled switches via main panel breaker
Updated to the latest Platform Version
Rebooted the hub a few times
Shutdown the hub and removed power for ~30 minutes
The makeup of the switches impacted is as follows:
Thank you for the reply! Just got back home and am compiling what you've requested. I'll do my best. My apologies in advance if I don't totally understand how to deliver exactly the right answers.
Ok, see all the ones with blue in the 01 row and columns? Those are the only devices where the hub and device think they can talk directly to each other. All the other devices would need to route through those devices to reach the hub. Does that seem correct? You can match the hex node ID up with the device in the details list.
Also the zwave details you sent me in a PM, you either did not refresh the stats or it was taken shortly after a reboot I think? All the RTTs are blank. I guess maybe that would make sense if the radio is totally locked up.
For Zwave it looks like you have the radio firmware updated but are still on the 7.15 SDK. You can follow these steps to be sure its all updated. The Update Firmware button will be hidden if the radio firmware is already up to date. In that case just run the gateway update endpoint and reboot.
Thanks for the explanation of the device topology. I think graph tracks. My hub is in a network cabinet, inside an a/v closet. Devices 07 and 0F are Z-Wave switches that flank that closet. So, it is not altogether surprising that those would be the most leveraged final hop in the routes.
As for the RTTs, I did press the "Refresh Statistics" button on the devices page prior to grabbing that screenshot. At that time, it had been about 16 hours since a reset. After i saw your comment, I pressed it again to be sure. One device popped up with an RTT. I pressed it again and a few more populated. I pressed it several times and I ended up with this:
I will say that, outside of this specific issue related to the reporting of the Current State of the physical switches, the Z-Wave Network is very reliable and has been for a few years.
I will work on the updates you described in your last paragraph now and report back.
I tried updating the firmware on a Leviton switch to 1.22. That was harder than expected. I ended up using your advice to change the file type and finally got it to work.
Looking in your details list, I see a lot of devices trying to route through things that no longer show the hub as a neighbor. This could be where its getting all jammed up.
Keep the hub outside of the cabinet. Try running repair on just node 0x0007. Then you could also try running it on a few of the other devices near the hub that should work as repeaters to the hub. Wait a few minutes and refresh the stats. See if the topo map shows any new blue marks on 01.
The other thing I see in the other details list you sent me is 3 instances of a USB stick (PC_CONTROLLER), and one other ghost node. You should try to remove these if you can. How To Remove Ghost Devices using hub tools or a UZB Stick <-- Check Section 3
Just to make sure it wasn't a fluke, I went to one of my currently installed switches and ran through the same (exclude, reset, include) process.
That switch also now works as it should.
I will try running a repair on node 007. I will also remove the extra USB sticks and the ghost node (that one has been particularly tricky to get rid of). Those are all matters of good measure and should be done not matter what.
Does the revelation of newly paired nodes functioning as designed spark any idea for another solution?
It reinforces the need for the repairs to try to get everything working again.
Possibly something bad was going on for over 24hrs, and the mesh was trying to self-heal. This can actually make it worse if neighbors are unreachable due to some sot of issue going on. The healing may break a lot of the neighbors if they are not responding. Once you get to that point it can take a bit to recover naturally.
If you can get the main repeating nodes repaired and talking to the hub correctly, then you might be able to run a full repair to kick start everything else getting back to normal.
I tried a few things. Nothing worked. Removed the ghost nodes as suggested. Then, ultimately, I ended up manually rebuilding the route for all of the nodes that were the last hop in another node's route - one by one. Then, I tried "Rebuild Network". That took a long time.
The same issue persists. Any other ideas or diagnostics to try?
Which is, NO devices report state changes back to the hub? Or just some devices? But if you command the device from the hub, that works and reports the state change back? Is that correct?
I think we may have been going in the wrong direction, however your mesh does look a lot better now!
Try out my Leviton driver for the dimmers. I think some of those old Leviton devices send a HAIL command and a lot of the system drivers do not handle it correctly. This driver is built for the Leviton devices: [DRIVER] Leviton Dimmers (DZPD3 / DZ6HD)
You can also switch one of them to the "Device" driver, enable debug if its an option. Then try activating the device and see if anything gets logged. Post a screenshot of the logs.
Depending on those things, I have one other idea that the lifeline association might be lost which would be very odd. If you turn on debug logging on my Leviton driver and then run the Configure command, it should try to set the Lifeline association if its missing, so you can try that also. The debug logging will show you in the logs what it did exactly.
Correct. No devices with a physical switch report the state change when the physical switch is used to change the state. UNLESS I exclude and re-add, then it works fine.
Correct. If the command is sent by the hub, everything works as expected.
I have 4 other C7 hubs paired with the same models of switches that are NOT experiencing this problem.
Thank you for sending over your drivers. I will experiment and post screenshots of the logs shortly.
Holy s**t it worked! I checked two separate devices, just to be sure. You can derive from the logs (from the lack of "Physical" activity before) that it did require running the configure command.
