C8 Zwave randomly stops working

[aarony]

Hey since moving to the C8 zwave stops working randomly. All the devices are listed but none of them respond. If I reboot the hub then it says that the zwave is enabled, but it shows zero devices. I have tried to disable and re-enable the ratio, but that does nothing. I have to shut down the system, unplug it then plug it back in. Then everything works ok.

It looks like the zwave doesn’t have any firmware to upgrade. I’m running on 2.3.5.152 code on the hub.

Any ideas?

[csteele]

@aarony

I have tried to disable and re-enable the ratio, but that does nothing. I have to shut down the system, unplug it then plug it back in.

This is correct. Enabling/disabling is simply a software block between the Platform and the ZWave radio SOC. The Radio SOCs are powered 100% of the time so when they get "jammed up" the only resolution is a power cycle: shutdown, wait for red, unplug, wait 10 seconds, power up.

I have 2 C-8's running and yesterday when I looked, they had been running for 40+ days. In other words it's not ordinary for the ZRadio to "jam up" over such a short time frame. I'd suspect mesh over consumption... and the usual examples of this is power monitoring. It's "normal" for vendors of products that also include power monitoring to set the default reporting interval very small. It's sort of a marketing gimmick in my opinion.. you add the device and you can see that power is reporting very quickly. The only way to do that is to send a report every second. If that's the only device you have, then who cares, right? But Hubitat Hub users rarely have only one and that high reporting rate (low reporting interval) is harmful. Logs should have some hints.

[aarony]

Is that the fix? Just do a hard reboot?

[mavrrick58]

@aarony

Is that the fix? Just do a hard reboot?

If the Zwave Radio is truely gumed up yes, but i think more importantly what @csteele is pointing out is that there is likely a cause for your radio getting hung.

With that said. lets ask a few questions to get the bigger picture

1. How frequently is this happeing. Daily, weekly, Montly,
2. Can you post your Zwave Mesh details page so we can see how it looks when it is working
3. When the radio hangs up does anything else work, can you get to the diagnotistic page.
4. Do you know what the memory usage on your hub is like?
5. What is the mix of zwave generation devices. ie 300, 500, 700,800
6. To go with what @csteele said what kind of devices do you have and do you have anything that reports frequently like the power devices he mentioned.

[aarony]

I’m going to do my best to work through these questions. I’ll gladly provide more if you need it.

1. How frequently is this happeing. Daily, weekly, —Montly, Monthly…I did read that it does happen when a cloud backup fails. I did have that happen today. I also just realized that my backups and weekly reboots were scheduled at the same time. I have made adjustments to those.
2. Can you post your Zwave Mesh details page so we can see how it looks when it is working —I will post this at the bottom.
3. When the radio hangs up does anything else work, can you get to the diagnotistic page. —I didn’t try any zigbee devices, but I had some local WiFi devices that connect via an app. They worked flawlessly.
4. Do you know what the memory usage on your hub is like? I have attached a screenshot below that includes the free memory.
5. What is the mix of zwave generation devices. ie 300, 500, 700,800 — I have no idea. I have a lot of older wave devices that I had setup on Smartthings and brought over about 3 or 4 years ago. Plus some I’ve recently added if you tell
6. To go with what @csteele said what kind of devices do you have and do you have anything that reports frequently like the power devices he mentioned. I have a WeatherStation the reports every 5 minutes…that connects via API call. No zwave device that reports more frequently than that. Most of them are 24 hours or when a sensor is trigger or switch changes.

I hope that helps.

[mavrrick58]

Monthly may not be to bad if your hub is very busy. Looking at your Zwave mesh image it does seem to be pretty busy .

I was actually thinking about the Zwave details page and not the mesh graphic. I was looking for ghosts, but that does look like you have a few devices disconnected though. Are those devices not used. You may want to exclude them if they are not plugged in to contribute to the mesh.

You may want to put a RM rule in place to notify you if the Zwave Radio Crashes. An example of that is below. Not much to it.

The image for the HubInfo device driver just shows momentary information for memory usage. What would really be helpful is to know what it is like just before the zwave radio crashes. Low memory has been found to be related to radio crashes. You can track this a few ways, but I use Influxdb Logger, a InfluxDB, and Grafana. You technically don't need a always on computer as you can get free cloud accounts for InfluxDB and Grafana.

This Grafana dashboard shows my hub's current state over the last 2 days. My memory on my prod hub starts around 490mb after reboot, but then drops rapidly for about 24 hours and then starts to level off.

From my experience very older generation hardware can be problematic. If you have anything that isn't Zwave Plus(zwave 500) I would suggest working on replacing it.

Last thing you may want to do is enable reboot and the HubInfo driver and then setup a rule that will do a reboot if the freeMemory metric gets below 100MB or so. Here is another rule I wrote for that.

[aarony]

So it’s roughly a week later and it happened again. To note I also had a failed cloud backup happen also. Sorry I didn’t setup up memory monitoring yet. I didn’t setup the zwave monitor. It did exactly as you said. This time a simple reboot solved the issue. I can setup a low memory reboot. I will say this every time this has happened it’s been in the middle of the night when devices aren’t turning on and off, and other timed events are happening.

Here are the constants: failed cloud backup and in the middle of the night.