Zigbee Storm

I had an interesting experience today with Zigbee.

One of my C8s is a device hub, and operates pretty much only my Zigbee and Zwave devices. My other C8 runs a much smaller number of devices that are obscured by a lot of ducting and piping in my basement. The 2 "device" hubs are separated by about 20 ft (and a fair bit of metal in the line of site path).

Anyway, I happened to be sitting upstairs today, and heard one of my Zigbee switches click. Then another, and then they repeated at random times fairly frequently.

Checking my logs, i wasnt surprised to find the Zigbee network going up and down. I restarted the zigbee radio, ran the rebuild....and the relays continued clicking. I then turned off the hub. And they continued. Huh?

After a few minutes, they settled, i restarted the hub, and all has been well.

Not sure what to think. Since I've had my devices on the C8s, this has happened off and on, but despite my efforts, i've never isolated the problem.

However, i'm really beginning to suspect a malfuntioning device. The Hub seems to be out of the picture since the behavior continued when the hub was powered off.

No idea how to isolate it either.

Just thought I'd share, in case someone else experiences something similar.

S.

You win the prize for strangest problem report of the week, I mean month, no, I mean year. :wink:

You say you have two hubs but it appears you only turn off the hub w/the devices when this issue has occurred. Are your devices shared from your "device" hub to your other hub? Shouldn't matter if the source hub is off, but just thought I'd confirm.

Thanks! I'm so proud!

Yeah, the devices that were misbehaving were all on one of my two "device" hubs. And I only shut that hub off. That hub does share to my other hubs over hub mesh -- as you noted. However, post-facto, I realized that the problem may have been one of my Samsung Buttons (Zigbee).

I noticed recently that I had changed the button's battery twice in recent weeks, and that the button on the other side of the room hadn't been changed at all, and hadn't dropped but a percent or so in Battery. According to my battery reports the "bad" button reported itself at 82% in the morning of the "Storm", and the "Good" button remained at 90%, by noon on the same day as the "storm" which occurred around 10 AM, the bad button was reported to be 42% which in my experience with SAMSUNG buttons is the same as "DEAD".

"Bad" button is in relatively close proximity to the devices that were misbehaving, and something drank all the voltage goodness out of that brave little button cell, so I'm guessing "bad" button lost it's poor little Zigbee mind, and spammed the network with packets.

Of course, I also realized that the lack of my Hub, doesn't constitute the lack of a Zigbee mesh, it just means that it has not coordinator/controller. So turning off the hub, only protected the hub radio from a whole bunch of resets.

If my speculation is correct, once the battery died on the button, the Storm stopped, and everything was fine -- I replaced the button, and it's on it's way to the Smart Device Drawer of Shame...

I'll check battery levels and hub logs when I get home today -- we'll see if the Hub has suffered any more Zigbee "Storms" or outages.

Scott

1 Like

Wow...lots of fun in the big city. Could indeed be related to the issues you mention, so that looks like a good path to wander down.

Don't worry, you get to keep your title and we'll be sending you a T-Shirt commemorating the moment in a couple weeks. :wink:

1 Like

You aren't the only one to have a ZigBee on/off situation. It's been happening for the last few versions. Check your location logs.

Yeah - honestly, that's why I posted in the Lounge this time. -- I pushed one of the early reports, not long after the C8 came out. I had thought (then) that it was caused by one of my plugin-outlets. I removed some devices, (then) and the problem was much diminished. My recurrence has been somewhat random since then. Note, it was also speculated that Zigbee Mesh congestion (too many devices) might be the cause -- I don't think I have a particularly large Zigbee Mesh (sub 70), but that seems a possibility too, or possibly just something that exacerbates the problem.

Having said that, before, it was one or two clicks of the relays, a pause for some random period of time, and then a few more...and in the location logs, I noted the a couple of corresponding up-downs of the radio. This time it was a veritable storm.

I've gone a month or so without any events, and then a few, and then none again. Nailing this down is going to be hard unless we can establish a consistent cause or causes.

Personally, I think this is a device issue -- why the C8's are sensitive to it, and the C7's and C5s, C4s, not so much? I have no idea. Perhaps something in the older Zigbee stacks was dropped in the newer radios?

Scott

1 Like

For the record, it has now been more than 30 hours since I removed the Samsung Button of Zigbee Storms (tm), and my Hubitat hub has been stable, with no Zigbee storms.

S.

2 Likes

WOOT! I guess you know what you're getting for the holidays...a stable hub. :slight_smile:

1 Like

Fingers Crossed. :crossed_fingers:

1 Like

Welp. So much for that idea.

I had a single cycle online/offline today. Just one.

I think im going to have to get an Xbee and try to capture one of these events. I'm fresh out of other ideas...

S.

Was there any effect on your Zigbee devices - any device drop-offs?

As long as you're going to do more investigations you should update to .142 - doesn't directly address Zigbee offline/online events, but usually best to be on the current release.

No idea. I just saw it in the logs--I'll do a little digging before I update. See whether there are any indicators....

Hey! How'd you know I hadn't hit the .142 download yet? Creepy @danabw creepy. Lol

S

Just coincidence, nothing special.

Oh, and BTW, your fridge door is open, and looks like it's time to stock up on some more beverages. :wink:

1 Like

Good point. That damned fridge door.

Anyway, the only thing I noted was that my new Tuya 2 in 1 motion sensors are a bit chatty (Tuya 2 in 1 Zigbee Mini PIR Motion Detector + Bright Lux ZG-204ZL) they seem to push tons of loggable messages out. That could be adding to the problem i suppose.

Pure conjecture though. Not readily testable short of removing them.

Hm.

S.

@kkossev has created a lot of improved drivers for Tuya devices, if one is available from him for your device you aren't using it, you should do some searching in the forum to see if one is available to try. He typically provides options to ignore or suppress excessive/spurious logging from devices.

Yeah, im using one of his.

So. About that update. Triggered a storm.

I have two outlets relay chattering at me now.
...oh, make that 3.

Crap. Ok, Im going to start removing Zigbee devices, and see where we get.

Arrgh.

S.

1 Like

What brand/model are the outlets? If Tuya we may want to get some input from Kkossev.

The outlets that seem to chatter during one of these storms are the GE/Jasco in-walls and the Legrand in-walls. Frankly, i suspect others are behaving similarly, but their relays are just quieter.

I removed three plugin outlets in total so far. 1 Samsung older gen, from the other hub (just because). Then I removed a Sylvania (quite old, possibly ~2016 or 17) and a recent 3rd Reality, both from the hub now known as Mr. Stormy. I also lowered the Radio power to 8.

Something soothed the storm, and the radio is up and seemingly stable.

I'm pretty sure that I very likely only have 1 or 2 older Zigbee plugins left installed, and thats on the other device hub. The main device hub only has 1 or 2 of the recent Third reality outlets and I want to say they are Zigbee 3.0. (3RSP019BZ). Confirmed. Zigbee 3.

I have quite a number of older Samsung buttons and motion sensors though. I'm really beginning to wonder if these older devices are causing issues with the newer Zigbee 3 stack/radios.

Ha, now everytime i hear a little click in the house, I wonder if it's a relay!

S.

1 Like

So. On the .143 update, when the hub came up, there was a lot of relay clicking, and I was able to confirm it also affected the light switches.

Some device must get crazy spammy and blast the mesh. I really don't think this is the hub.

S.

1 Like

Enable logging on devices & apps (including debug) and you should be able to find the culprit(s). :slight_smile: