Zigbee mesh is going crazy after years of stability - help

Quick note: after putting the batteries back in, the devices resumed spamming the logs. The pattern articulated above, however, remained consistent even for the battery powered devices. Profile 0x0, cluster 0x6, followed by 5 rapid instances of profile 0x104, cluster 0x19

i am getting the exact same thing but my mesh is working .. not sure that is abnormal and if it is, may be related to the firmware updates?

ie

1 Like

Iā€™m sad to report that this did not resolve the issue.

I powered off the hub for 60 minutes. When I turned the hub back on the spam resumed within 1 minute of rebooting.

Next step is researching how to roll back to 2.3.2 firmware to rule out 2.3.3 being a root cause.

1 Like

:disappointed: Not much research needed. Go to Diagnostic Tool and click Restore Previous Version, then select 2.3.2.141.

2 Likes

Indeed, I just finished restoring from my last backup before 2.3.2 and have something interesting to show you and @mike.maxwell .

It appears that I had an issue before 2.3.2, but that 2.3.3 made it worse.

Here are my logs after rolling back to 2.3.2.141, and then restoring from a backup I made on Monday just before I had updated to 2.3.3 previously.

Note that things are much quieter, and the hub seems faster overall (quicker response when clicking on buttons, etc)

There still is spam, but only 2 messages per device rather than 5.

With Firmware 2.3.2, each device shows Profile 0x0, cluster 0x6 followed by a single instance of profile 0x104, cluster 0x19 for each device. This is an improvement from the 5 rapid instances of profile 0x104, cluster 0x19 for each device that we saw with 2.3.3.

Screenshot of 2.3.2 logs below that can be compared against my screenshot of 2.3.3 logs above.

Summary: I'm no expert, and something is causing interference in my Zigbee mesh. The impact of my interference appears to be exacerbated by something in the 2.3.3 firmware update.

A post was split to a new topic: Can more expensive hardware make Zigbee better?

Final update for a while: For the past hour, things have been a lot quieter on 2.3.2.

Here's the latest screenshot. Note that the log messages are significantly less frequent, and smaller in number.

On 2.3.3 there was a steady stream of Zigbee spam, with messages every 2-3 seconds. Sometimes multiple messages per second. Each device would send 6 total messages per 'spam' instance. Thus, more frequent spam, and each occurrence had 6 instances of zigbee messages per device.

On 2.3.2 the Zigbee spam is much less frequent. About a message 30 seconds to 2 minutes. Each occurrence has just 2 instances of zigbee messages. So, less occurrences, and less messages per occurrence. Overall the hub seems faster and more responsive when navigating the UI on 2.3.2 since the rollback.

After all the debugging and tests, it appears that something in 2.3.3 caused a 3x+ increase in zigbee spam throughout my mesh. I am not saying my mesh is perfect, clearly, there is some interference, but 2.3.3 seems to have made my situation much worse. I'll be staying on 2.3.2 for a while until I hear that a root cause is determined/fixed.

Here's the latest screenshot of my Zigbee logs:

Thanks to all who offered suggestions and condolences while I debugged and tested. I appreciate y'all!

1 Like

Are you in the beta-test group? And, if so, had you applied any changes related to the zigbee radio during the beta test of 2.3.3?

3 Likes

Nope and nope

1 Like

Tagging @support_team and @mike.maxwell

Following this thread, I must admit that I hadnā€™t looked at my Zigbee logs during the beta. Iā€™ve not had any issues except an occasional slow ML rule thatā€™s a new issue. Looking at the logs, Iā€™m also seeing a lot more Zigbee traffic than I did prior to 2.3.3. Itā€™s so much traffic on my mesh thatā€™s mostly end devices ( over 80 Zigbee devices) that Iā€™m really surprised I havenā€™t had bigger problems.

3 Likes

Same here lots of messages. But not sure that is abnormal.

1 Like

For me, the fact that so many sensors were sending messages this morning when everyone was still in bed was very strange to see. They are usually silent at that time. I just checked again (everyone is in bed including me) and itā€™s a constant stream of messages in the Zigbee log. Iā€™m not sure what to make of it since everything is working fine.

1 Like

This is what I was seeing as well when on 2.3.3.

A few weeks ago I added a couple Sonoff S40 outlets, and then over this weekend noticed a couple times that one or two bulbs didnā€™t turn on properly. I assumed the Sonoff outlets had caused a mesh problem, and went through the steps I outlined in my first post.

After I was on 2.3.3 more lights and devices were falling off the network more frequently. Initially just power cycling a light switch was enough to get it working, but by Wednesday this week just a power cycle wasnā€™t enough. To get devices back on my mesh I had to reset and re-pair them. This happened to 2 Fyrtur shades and 2 Sengled Classic bulbs that hadnā€™t given me issues in a few years.

My final step of debugging was rolling back to 2.3.2.

Now that Iā€™m back on 2.3.2 things have been stable for 14 hours. No devices have fallen off yet.

My (probably wrong) hypothesis is that my mesh already isnā€™t the best due to RF interference, but had been ā€˜good enoughā€™ prior to 2.3.3. However something in 2.3.3 changed and compounded the impact of my mesh issues so that devices were falling off and failing to activate more frequently and WAF took a dip.

2 Likes

I'm seeing the same thing, traffic wise (though without a noticeable performance impact) having upgraded my C-3 to 2.3.3.125; I started a new thread since I don't think it's related to a mesh stability issue or noise.

What's up with all the extra Zigbee OTA traffic on 2.3.3? - :bellhop_bell: Get Help / Devices - Hubitat

9 Likes

For those following this thread, the root issue has been found and will be fixed in one of the next releases. Thanks all!

10 Likes

Please try 2.3.3.126, just released.

1 Like

Awesome!!

I tried to update and was able to get 2.3.3.125, but can't get 2.3.3.126 to show up. Did you guys pull it, or is there anything I need to do on my end? I'm hitting Settings > Check for Updates. I've rebooted the hub as well and tried again to no avail.

(I updated to 2.3.3.125 because I thought maybe I had to do that in order to get 2.3.3.126 to show up :stuck_out_tongue: )

.126 was pulled.. .127 presumebly will be out soon with the fix.

3 Likes

It was pulled.

1 Like