Complete Failure / Hub Malfunction?

🛎️ Get Help Devices
, ,
1

As of today, Both Zigbee & Z-Wave devices are now unresponsive, statuses are incorrect (locks state unlocked when they are locked), get random "Device has opened", when right in front of it and it hasn't changed physically.

I performed the soft "reset" this morning by powering down, unplugging and leaving the Hub sit for 5 minutes. Rebooted, came online, the Hub is responsive, but my devices still show incorrect statuses and won't respond to commands (Zigbee & Z-Wave).

  • Hardware: Rev C-7
  • Platform: 2.3.1.142
  • Static IP on the LAN
  • No software updates for a few weeks
  • No new Apps installed recently
  • No new Zigbee or Z-Wave devices added in since Dec 2021
  • Moved 1 Z-Wave repeater 3 feet 2 weeks ago. It was and still showing online and communicating via Z-Wave Details
  • All Device batteries replaced in Jan - Most were showing over 80%, none lower than 50% as of last weekend.

I restored settings to 5-28, prior to issues starting, no difference in how things are responding/reporting.

Details: I noticed on Wednesday that one of my Sengled Zigbee bulbs wouldn't turn on from Zooz switch. Just figured I would deal with it this weekend as it was just in the closet. Thursday evening, doors/windows in Devices and dashboard were showing they were open when they weren't. Had to manually turn off lights via dashboard, but things did respond.

Yesterday as the day progressed, more and more devices became unresponsive, both Zigbee & Z-Wave. Our Presence was being trigged back and forth home/not home, varying windows & doors would alert being opened when they hadn't been physically. HSM wouldn't set due to so many devices within the hub showing open when they weren't.

Rule Machine Rules won't execute, or only portions of them. None of the physical Zooz switches are able to control anything, sometimes from the dashboard I can get a response, but that is random.

The hub is responsive, not lagging. Can move through the GUI and Dashboards without delay. I reviewed the logs last night, one app had an Error, WiFi Presence seemed to have more than normal /# of entries, so I disabled the app & virtual devices to see if that would help. Otherwise nothing else stands out being an issue.

The hub does not appear to be overloaded:
Running jobs: 0
Local Apps: 1m 17s busy
Devices: 0m 23s busy

Have I had a Zigbee/Z-Wave or hub Failure? Is this a software issue? Can I re-install the software without a full reset?

Any other thoughts or suggestions appreciated, feel like my left hand is cut off right now.

2

I don’t recall seeing a similar issue, but if I encountered this, what I would do at the point you’re at is perform a backup and save it locally, then a soft reset via :8081 and then restore the backup.

3

I suspect you have a failed device within your network. I've recently had a z-wave switch fail and cripple my entire system. I had no idea that one failed device would cause so much havoc.

6 Likes
4

@mdadkison

I have done a local back up my PC. I'll attempt @Sebastien recommendation, if that doesn't do it I'll start investigating all devices, unpair/re-add to see if that will stabilize things.

Is there another place in the logs past turning on debugging for each device (I have over 80) to see if I can narrow down the device that might be causing this?

Just a packed weekend, I won't have a good amount of time till Monday to really dig in.

Appreciate the replies.

1 Like
5

The Z-Wave Details screen can Help determine Z-Wave issues. You can add a screenshot of yours here - there are quite a few people in the community adept at reading those. This won’t help with Zigbee however.

On the Zigbee side, issues can sometimes be caused by having too much noise on the channel used. Chanel 20 is typically recommended.

As I write this, I am reminded that sometimes a Z-Wave device can start spamming the Z-Wave frequencies. One thing you could try is to reboot your powered Z-Wave devices. The easiest way to do this is to power down the relevant breakers, or just power-off the whole house (That’s how I did this in the past.). If you do this, be sure that your hub is shut down properly or on a battery backup first…

1 Like
6

Anything with excessive use in the App Stats or Device Stats in the Logs section? I usually open the settings on each of those pages and enable all stats to get a better picture as to what is going on?

:+1:

1 Like
7

This

8

Thank you again for the replies and suggestions.

Here is what I have done, things are starting to stabilize:

  • Enabled Debug logging on devices
  • Found a relay was logging a great deal of errors, unplugged it - Things started to get better, some devices started to respond but not 100%.
  • Today I unplugged/remove batteries from all Z-Wave devices; Started process of removing the ghost devices - Successful in all but 3. Tried all suggestions from the community, I don't have a z-wave stick, so was relying solely on the HE utility.
  • I shutdown HE, plugged all of my devices back in.
  • After HE was running for 30 minutes, I started a Z-Wave Repair - 1/2 of my devices failed - More cleanup/repair/remove re-pair work to be done. Or is this just that the Z-Wave radio to needs to stabilize after the shutdown?

I'll see how things are after this. If I still have issues, I'll order up the Z-Wave stick to work on getting the last Ghost devices removed. I understand this is not a Hubitat issue, but would really like to see a better way to deal with these, I know progress has been made.

Hopefully my next update I can report that things are back to normal.

9

It would be nice to see your Zwave Settings page here.

You said you tried everything, but did you try any of the following?

  • After rebooting, immediately attempt to remove one of the ghosts. Don't do more than one and reboot again.
  • Did you find (or think you found) the device that caused the ghost? Did you de-power that? It is usually the next 1-2 items below the ghost device. Line powered devices might need to have the circuit breaker tripped on that device.
  • Did you try wrapping the hub in foil, and see if that attenuated the Zwave signal enough to do a ghost removal? Long shot, but I heard it works sometimes.

It is not generally recommended or necessary to do a full Zwave repair. IF you have to do a repair, do individual nodes.

Unplugged, or unpaired from hub? Simply unplugging might actually cause issues if things are routing through that device.

Did you do the Zwave radio firmware update when you did the last hub update?

1 Like
10

Any left will keep things lagging and your mesh a mess. The devices that created the ghosts need to be powered down... Then hub off (pull power from wall only never from hub) for 5 mins and brought back up before attempting to remove again.

Post your z-wave details page for us...

1 Like
11

I have 1 line powered device that I disconnected power to before trying the Remove, all other devices I unplugged or took batteries out.

I did do a firmware update earlier this year, here is the results from the URL with my hub IP:

VersionReport(zWaveLibraryType:7, zWaveProtocolVersion:7, zWaveProtocolSubVersion:17, firmware0Version:7, firmware0SubVersion:17, hardwareVersion:1, firmwareTargets:1, targetVersions:[[target:1, version:7, subVersion:15]])

I compared my Z-Wave Details page to the image you sent, I don't have a Firmware Update button. I recall that being there before, but is not present now:

image
image1909Ă—327 39.2 KB

I attempted a repair of the z-wave multi-relay that I initially unplugged (logging all the errors), it wouldn't complete. I then attempted to a replace - failed. Tried an exclude, that failed as well. I attempted unsuccessfully to repair the individual devices that were running through (using the Z-Wave details page) that device. I had never done a full reset till today, always attempt an individual repair first.

I have since excluded (some were successful, some not - even moved some closer to the hub) and re-added the Z-Wave devices that were running through the unplugged device, but left with another ghost. The new device instances are now hopping to other nodes, not the relay that I removed. No devices are showing online that are hopping through that device.

I will have to get a Z-Wave stick at this point to remove the ghosts and get this fully cleaned up, after multiple attempts refreshing, clicking Remove, shutting down the hub etc, they just won't Remove using the utility.

  • I attempted the remove immediately after reboot, I can try again.
  • I did not do the foil trick, I will attempt that later

For now, things are OK, much better than on Saturday, more work to be done. Thank you for the suggestions.

12

Post your z-wave details page in it's entirety. We can track things better that way (there is no security issues with doing so)

2 Likes
13

@neonturbo
@rlithgow1

Just got back into town, and able to work on it a bit more including getting the screen capture.

Attached is the Z-Wave Details page in full:

Z-Wave-Details
Z-Wave-Details1873Ă—4888 1.26 MB

Things go from stable, to not, like yesterday evening, Zigbee devices were fine, but no Z-Wave would respond. I know the ghost devices need to be dealt with.

Device 0x46 has a route, but that device ID does not exist in Devices. The other Ghosted devices I have attempted to remove after doing a refresh/repair respectively without success.

I am not sure if this is normal, or if it does point to a Z-Wave radio issue, but at times I can open the Z-Wave details and all Routes are blank. I can refresh/update the page, but they stay blank. Only way I have found to get the routes to display is by rebooting the Hub.

I did get a Firmware Update button today on the Z-Wave details page, clicked it - Z-Wave updater: Firmware is already up to date.

I'll have more time this weekend to power things down again and attempt to remove the Ghost devices directly after reboot as @neonturbo suggested.

1 Like
14

You have at least 3 legit ghosts in there. For the other stuff without routes for now, shut down the hub, unplug hub for 5 full minutes (at the wall not the hub) and bring back up and see how it looks. Get rid of the ghosts, re pair anything with security without it (except for door locks and garage door stuff). Your mesh is a real mess at the moment :frowning:

4 Likes