[2.2.4.156] Z-Wave offline

Background: I bought a new C7 during the Thanksgiving holiday, have had issues migrating some of my old GE Switches, so not everything is moved over successfully

Previous hub is a C4. It is turned off.

I updated to 2.2.4.156 yesterday at around 6PM. Quick tests suggested things were normal.

This morning the C7 can't control anything Z-wave. Z-wave log shows commands going out, but none of the devices are responding. Rebooting did not help.

Z-wave repair listed every node it attempted to contact as failed. Screen Shot 2020-12-05 at 11.01.46 AM

Any thoughts before I roll back to 2.2.4.148, the only other version I have available on my C7?

Did you try a clean shutdown to red light, power down for 30 sec., power back up? That would reset the Z-Wave radio.

Update: Not sure if it's a caffeine deficiency but I didn't think to power-off reboot the C7. I did that and now it's back working on Z-Wave.

Suggestions for diagnostics in the future are welcome, if I can help gather input on what might have gotten hung.

Reply just crossed, that seems to have reset it. What can I do to help diagnose in the future?

I’m not an expert, but, when you’ve got a complete failure of all nodes, this is about the only thing you can do.

The spiffy Hubitat Z-Wave Mesh Details app (also available for install in Hubitat Package Manager) does provide some insight if some things are up. Aside from that, only a Z-Wave sniffer can really tell you what is happening.

Well, that’s a problem that is easily solved.

Thanks for the tip on Tony's Mesh tool. I already loaded the awesome Package Manager, and it loaded the tool up.

The radio stayed up only until I ran a Z-Wave Repair, than hung again. I reset one more time, tested a bit, then once again ran a Repair and the radio hung about 3 devices in.

I know the classic response to "My arm hurts when I lift it like this" is "So don't lift it like that", and I guess I can leave the network alone for now. But it would be nice to know what's happening, and if I should expect to keep resetting my device.

N.B., as it's a very new device, I'm open to the idea that it's an out of box hardware failure.

More likely an out of box firmware failure. If you have 2.2.4.148, try rolling back. Do you know how.

I haven't done a rollback before, but I see the option on http://device-address:8081

Is that the preferred method?

Yep. Backup first.

To do the rollback, you have to log in to the Diagnostic Tool with the MAC address of the hub. Easiest way is to go to Settings, Hub Details, copy the MAC address, so that you have the correct format, then paste into MAC address field for Diagnostic Tool login. It will log you out after a short time (few minutes),

Thanks, I rolled back successfully. I was even able to lift my arm.. er.. run a Z-Wave Repair without any issue.

Before I rolled back I tried the "update firmware" button on the Z-Wave page, which reported that I was up to date on firmware. Is there any other verification method for this?

Or should I just wait for the next release, watch the comments to see experience of other users, then update as I need to later?

The button on the Z-Wave page doesn’t do what you seem to expect it to do. It has nothing to do with the platform firmware (e.g., the 2.2.4.x firmware).

The Z-Wave Details page updates the firmware in the Z-Wave Radio. Those of us (such as me) with initial manufacturing versions of the C-7 received a version of Z-Wave Radio firmware that was not, shall we say, optimal. Hubitat is the first hub to be Z-Wave certified with SiLabs 700 series chip. The Z-Wave Radio stack software is from SiLabs, not Hubitat. While the 700 series Z-Wave chip has many nice features and improvements, there are still many bugs in the Z-Wave Radio stack, and those issues are what Hubitat engineering is working around in the recent flurry of platform releases. In parallel with this, Hubitat engineering has been working to maintain a common platform code base that runs on the C-3, C-4, and C-5 that have a different Z-Wave radio.

There has, to date, only been one update of the Z-Wave Radio’s stack. Whether you have the current up-to-date radio firmware is what that button checks (and fixes).

The platform software (e.g., 2.2.4.x) includes the built-in device drivers and built-in apps, the Hubitat Groovy platform that supports the hardware, etc.

Now, with that background, the only other validation method to check the radio stack firmware is to run the following command, and look at all of the version numbers of the various Z-Wave Radio Stack modules:

http://YourHubIP/hub/zwaveVersion

Here’s what it produces on my C-7:

One would need to know the correct module version numbers to decipher this output. That’s above my pay grade.

In the past, Hubitat staff has posted a note with the platform firmware release when it is necessary to update the radio firmware. As I noted above, that’s only happened once.

Many thanks again for the help. I'll keep an eye on update announcements etc. A long time ago I used to work in software QA (most of which was for dial up modem products, so yeah, a long time) but I appreciate the different components are owned by different orgs., and there's only so much you can do with a sourced component.

If the radio stays stable now (so far so good) it moves it off my hot list for the weekend, and I can worry about it more another day.

Cheers!

1 Like

FYI, here is what the notice looked like for the (only, to date) Z-Wave Radio firmware update. See the Important Please Read note.

I migrated from a C5 to a C7 hub a couple months ago. No significant issues, I basically rebuilt my z-wave network and apps from the ground up if I recall exactly.

I added indoor NEO z-wave plugs and outdoor GE switches over the past couple weeks for Christmas lights/decorations, and noticed most of the devices stopped responding to commands. Seems slightly different than the issues described here and in this post:

In my case some devices continued to work fine, z-wave mesh network tool showed some devices connected, but most of the mesh completely broken. I tried individual z-wave repairs, then an entire network z-wave repair which resulted in most of the nodes lost contact.

What was interesting is that commands to the device did not register, however when I manually changed the device, the status immediately updated on the device page. It appeared that communication from the device was working fine, but communication to the device was lost or simply not present.

Hub reboots did not help, however I'm trying the shutdown/restart to reset the z-wave radio. Hopefully that helps.

Additionally - I tried excluding and re-adding some of the problem nodes, the devices were simply not recognized by the hub in an exclude (or include) process, even when in the same room as the hub. I've never had that problem before, normally excludes work fine. It isn't uncommon for me when adding devices to first move it closer to the hub to initially add it to my network, then move it to the final location and cycle it/rebuild it a couple times to fix the final/best z-wave path.

I'm continuing to work through it today, I'm hoping I can get the network back working reliably without a z-wave network reset and re-adding all my devices (rebuilding apps/etc).

Another interesting detail, my z-wave logs are completely empty (from the Z-wave details page). I noticed activity on another C7 hub I'm running in another location, but nothing on my home hub. UPDATE - I'm seeing activity now, it appears the log needs to be open to view the activity.

I'm running 2.2.4.158 with the latest z-wave firmware update.

Your hub seems to be dealing with a significant number of events from various power/energy reporting devices, including the Zooz Double Plug . If you didn't do so already, please try to reduce the amount of power/energy reports that your devices are generating.

1 Like