Zigbee network no longer functional after update to 2.3.0.112

After updating from 2.2.9.146 to 2.3.0.112, my zigbee devices are completely unresponsive. There are no logs or alerts indicating any issue... In fact, the zigbee logs have basically gone to nothing, with a single entry every 5 minutes and none of the normal zigbee chatter. See if you can pinpoint the point where I did the update:

I've reverted back to 2.2.9, done a soft reset, done hard shutdowns, so far no change. And no info that I can see that help. Help!

Not clear what you mean by that. The only way that I know of to reset the radios is to shut down the hub, preferably, cleanly, as from Diag Tools (hub.IP:8081) or Settings, Shutdown, then turn off the power, leave it off for 20 seconds or so, then apply power and let it boot.

Now, it’s possible that the Zigbee radio module has failed.

It’s possible that the radio has changed Zigbee channels. Some have reported that.

But, if you power the hub down for an extended time, that causes Zigbee devices to panic and try to find the hub on its new channel when it comes back up.

Thanks, yes I'll elaborate:

In the past 6 months or so, I've had an issue with my zwave radio that was only recoverable by physically powering down the device (cleanly), as restarts did not reset the modem. So that's what I'm referring to when I say hard shutdowns.

I also had the growing database issue that was patched a few releases ago, which manifested when you did a hard shutdown, and would resolve itself if you did a regular reboot.

So I have had the box shut down for 60 seconds at a time, but no luck yet. No devices have panicked and come back available.

What I do see, and this could be completely useless, is that the zigbee logging since the original upgrade reports clusterId: 32818, where previous it was 6 and 25. No idea what those numbers represent, but they did change after the upgrade.

Alright, result... I tried turning it off... Waiting 10 minutes... Then on. But nothing.

I just manually changed the channels and I'm starting to see logging traffic from some of the devices. I assume the "up to 24 hours" thing is going to happen as the network heals. Disaster hopefully averted!

I believe it is 20 minutes. You need the devices to start panicking.

1 Like

@lairdknox, sorry if my post wasn’t perfectly clear.

The first paragraph of my post, with shutdown of the hub for 20 seconds, described the procedure to reset the radios, just as I said.

The last paragraph of my post, where one would shut the hub down for an extended time (i.e., twenty or thirty minutes or more), is the different procedure to cause the mesh’s Zigbee radios to panic, just as you (and I) noted.

Sorry if it wasn’t clear that these were two different procedures. Thanks for the clarification.

1 Like

I see said the blind man as he picked up his hammer and saw.

This is pretty frustrating. My Zigbee network has been about 10% working for the past week+ since the update. I've contacted hubitat support but they've gone dark after I responded to their question regarding what browser I was using... I'm gonna try and do one big post with all the info, hopefully readably, and maybe somebody here has some insight that might crack this nut for me.

Background:

  • I have a C-7
  • I have 141 devices, 48 zigbee and the rest z-wave, the vast majority powered
  • I use Node Red for my automations
  • I pull all Hubitat logs that I'm aware of and push them to Datadog, so I have logs going back about two weeks:
    image

When I initially noticed the problem (Zigbee devices not responding) I posted here, and further checked the logs and noticed that the Zigbee logsocket had greatly diminished logging since I upgraded to 2.3.0.112. The logsocket was emitting a single log event, once every five minutes, that looked like this:

{
	"id": "3361",
	"content": {
		"timestamp": "2021-12-03T16:30:29.142+00:00",
		"tags": [
			"source:nodered",
			"name:0000",
			"devicename:0000"
		],
		"host": "hubitat",
		"service": "zigbee",
		"attributes": {
			"lastHopLqi": 255,
			"destinationEndpoint": 0,
			"groupId": 0,
			"clusterId": 32818,
			"type": "zigbeeRx",
			"sourceEndpoint": 0,
			"sequence": 74,
			"hostname": "hubitat",
			"service": "zigbee",
			"profileId": 0,
			"name": "0000",
			"id": 0,
			"lastHopRssi": 0,
			"time": "2021-12-03 11:30:27.872"
		}
	}
}

Previously, the logsocket was emitting roughly 5 events per minute coming from random devices as they turned on/off, reported temperature, etc.

I created this thread and folks talked about channel changes, which I did... And got approximately 10 devices to come back online when I switched to channel 15 (from 20). I saw traffic in the logs and assumed that given that "up to 24 hours" the rest of the network would heal. I went away to Thanksgiving and came back that Sunday, and not a single new device had recovered!

I've now realized that if I take a brand-new device and try to pair it, Hubitat does not discover it.

I've now attempted to reset the Zigbee radio by going into Zigbee settings, typing 'reset' and clicking reset... The dialog box comes up for a while, and then it drops me back on that page... All the devices are still listed. I'm not sure if that's normal or if the reset has failed. There are no logs mentioning that a reset was started, was finished, or failed.

The only indicator that the reset may have worked is that the 10 devices that worked... Well now they don't. So I have no Zigbee at all. I still cannot add new devices.

  • I have done restarts.
  • I have done shutdowns, unplug for 20+ minutes.
  • I have soft reset the hub and restored (I know this doesn't touch the radio).
  • I've gone into safe mode and done a zigbee reset
  • I've gone into safe mode and done a channel change

I'd throw up my hands and say the radio failed, but the fact that at least some of the devices did work confuses me enough that I thought I'd post this again.

Anybody?

Did you ever update to the .113 release? @gopher.ny, Could this be related to my much less catastrophic issue fixed in .113?

I’ve found that devices respond within a minute of a channel change or they never respond.

1 Like

Yes, I updated to .113 on Nov 28.

Support responded that they'd forwarded the info to engineering and then basically closed the case, so I don't have any resolution or even any troubleshooting. Since I need my house to work, I ordered a new C7 and I'll attempt to migrate to that. It'll either work, or I'll have 141 disconnected devices and a small family mutiny, so fun stuff! Wish me luck!

Sooo fun story... I brought up the new hub, migrated the backup. Z-wave came across as expected, but Zigbee was still hosed! No logs, no ability to re-pair any existing devices or add new ones. Just as broken as before. Support was not able to explain the issue or issue troubleshooting, so now I have two non-zigbee Hubitat C7's.

I have since migrated my Zigbee network to zigbee2mqtt, retaining Hubitat for zwave at this point. Lots more debug info available in z2m, and has a very nice web UI. Running in a docker instance on a RPi4 with a Conbee2 plug.

Could you please PM me your hubs' ids? Bad radio happen sometimes, but two in a row is very unlikely.

1 Like