C7 - Z-Wave Control Issues (Random or extreme delay)

[tjquinn]

C7
Platform Version: 2.3.9.164
Z-Wave Version:
VersionReport(zWaveLibraryType:7, zWaveProtocolVersion:7, zWaveProtocolSubVersion:18, firmware0Version:7, firmware0SubVersion:18, hardwareVersion:1, firmwareTargets:1, targetVersions:[[target:1, version:7, subVersion:15]])
Powered by a PoE Splitter currently.

For the last few weeks (since the most recent selection of updates) I've been having terrible z-wave functionality... I, unfortunately, can't say what version started it... because I don't remember... but everything was working great, then I did an updated, stuff got kinda stupid, another update came out, so I did that. then I noticed that my z-wave radio firmware was out of date, so I updated that hoping to fix the issue... and a few more updates have come out, and I'm keeping on top of them... all of this to no avail...

What I'm seeing is devices commanded to do something, simply don't respond. the status doesn't updated, as if no command was given. If I go manually actuate the device, eventually the status might update in the hub.

I have tried shutting down the hub, pulling power for up to an hour, shutting off power to the entire house, for 30 min, nothing seems to help the issue. for about 10 min after the hub is booted I can control things seemingly fine, but then it all goes sideways.

I did happen to catch this in the z-wave logs today, almost like the radio just doesn't send the commands... but when it does, they go fine and the devices do what was commanded.

I have replaced the PoE Splitter, and I've tried powering it with a standard wall plug, doesn't seem to effect the outcome.

Does anyone have any thoughts on this? has my z-wave radio kicked the bucket?
I'm not sure what else to try... I've got a significant number of devices, so starting over would be a pretty big deal...

[jtp10181]

You missed the last step of updating the firmware, which is updating the zip gateway back to 7.18. That probably will not help the issue but you should do it anyway. Updating Z-Wave radio on C-7 hubs after version 2.3.8.122

Other than that, we need to see the full z-wave details page, you may need multiple screenshots.

[tjquinn]

Thanks! I didn't know I needed to do that last step.
Unfortunately, it has not resolved the issue...

Here's the screenshots of the Z-Wave Settings page

[jtp10181]

Looks like the Theater Rumble is offline and you have a couple of ghost devices (with Discover button shown instead of a device) you could try to remove. How To: Remove Ghosts using hub tools or a UZB Stick

[tjquinn]

Would that cause all these lines that appear to be failed transmissions?
it almost looks like the radio is commanded to do something, and simply returns that it did, when it didn't... when the line has actual data in it, the commands work in the real world... when the lines don't have actual data, nothing happens in the real world...

I'm trying the process to remove those ghosts, I'll have to find my z-stick... the hub processes don't seem to be working.

[jtp10181]

Not sure what all those lines are. What happened at 10:01? Looks like a lot of commands tried to go out. You would need to check what devices those are.

[velvetfoot]

I'll chime in mostly for moral support.

It doesn't take much to bring down the entire z-wave mesh: just one misbehaving device.

It seems that it might work for a little while after a controlled shutdown/power cycle, but then, the misbehaving device brings it down after what, 20 minutes, you said?

I've seen a lot of similar threads, but as I recall, there is no documented, organized, logical, nevermind easy way to find the miscreant.

A lot of times it seems that someone will suspect a device, for whatever reason, disconnect it, physically, since its radio would still be active if powered, and voila.

As I recall, people have had to do this with a number of suspects before the guilty party was found.

Perhaps in those "golden 20 minutes" you could find something in the logs that could give you a clue. After that...

Now, this is assuming you weren't fooling around with adding devices or messing with rules right before it started acting up. That would be a clue.

I've let the updates slip in the past, and recall it was hard to 'catch up', but I have a feeling it's a Z-wave device and not an update, and just a coincidence. But, I'm not an expert, and you were somewhat 'in the hole' when it came to updates.

Would reverting back to an earlier version even be possible at this point?

Listen to @jtp10181 and the other big brains.

Good Luck

[tjquinn]

Alrighty, so it's taken a few days for me to get to this, but here's what I've tried.

the Hub ghost removal didn't work, tried for a couple hours... no joy.
I was able to get an Aeotec z-stick paired... seemingly partially, and was able to remove one of the 3 ghosts... but the z-stick only went up to 82 in device numbers... not the 150 I need to see the other two. so I couldn't do that.

after doing that, it's made no difference. z-wave just goes unresponsive via the hub for a few hours, then poof, it'll start working for 15/20 min, then gone again for another few hours.

when it's working, it works great! all the automations run, switches that have multi tap work snappy and as expected...

I'm beginning to think my z-wave radio didn't apply the update properly and broke... but I have no clue how to fix that... perhaps @bobbyD can look in the magic backend and see if there's anything?

it is becoming frustrating when we loose control of everything... just not sure how to proceed.

I'm open to other thoughts too, except the one that says 'reset your hub and re-pair all your devices' I've done that before and it was brutal....

[velvetfoot]

Again, no expert here, but...

Again, it takes only one bad device, the hard part is finding it.

Barring absolutely no clues in the logs, which you would think there would something there, maybe you come up with some strategy involving de-powering devices.

For battery devices you could take the battery out of a device and see happens. I don't know if you'd have to do a shutdown/power cycle. You would think if you remove the logjam, things should start moving immediately, but I don't know.

For mains powered devices, I also don't know. Ideally, by device, I suppose. My toggle switches don't have anything to pull, as in an air gap, so maybe I would flip a breaker. In my case, that would cover several switches-I have a house full of them. Things is, how would things work, mesh-wise, afterwards?

Maybe flip all the breakers, except for the hub's which ideally could be on a battery. Then close one, wait 20 minutes, close another one, wait 20 minutes, until something breaks. Whether that's practical from a mesh aspect, I wouldn't know. You'd sure shake things up though.

Plus, if you redid everything, just because it's included doesn't mean it couldn't still screw things up. I think.

Let the bigger brains prevail though.

Good luck.

[jtp10181]

"Shed Lights" looks like it is hurting. Very long response time (RTT) and only 1 neighbor.

Is that device online, is it used in any sort of automations?
I would power it off if you can and disable the device in the HE devices list, to prevent any commands from going out to it. Had another case recently where 1-2 devices with a horrible RTT was bogging down the whole mesh.

[velvetfoot]

Jeff,

Would you do a shutdown/power cycle after depowering and disabling the suspect device?

[jtp10181]

Yeah probably not a bad idea to follow with that to restart the radio.

[tjquinn]

A few points of clarification

My Z wave mesh has been this way for at least two years now, with the only changes being updates to the hub. I had to redo everything when I upgraded my hub from a C5 to the C7.

The Shed Lights might not be having a great time, it’s about 30 feet away from the house. I will try shutting that off and disabling it to see what that does, it is used in some automations, just to make sure that they get turned back off when they’re used.

Everything was working really well up till this most recent update cycle where I did both the hub, and the radio though…

[jtp10181]

You can roll back the hub platform if you think that was the cause, but it probably wasn't. I would highly doubt the radio update caused any issues, every z-wave firmware update was an improvement over the prior, I went through all of them as they came out when I was using a C7.