What's up with all the extra Zigbee OTA traffic on 2.3.3?

[Tony]

Yesterday I updated my C-3 to 2.3.3.125 and I'm also seeing continual periodic bursts of non-productive Zigbee traffic, specifically involving clusterID 0x19 (as @craigspree reported in his thread, periodically five profileID 0x104 clusterId 0x19 messages .03 seconds apart, always following a profileID 0x0 clusterId 0x6 message).

NXP's Cluster Library Guide says 0x19 is the OTA upgrade cluster common to all Zigbee profiles; profileID 0x0 ClusterID 0x6 is "Match Descriptor Request". My guess is that devices are constantly looking for an OTA upgrade server, and doing it more frequently than necessary.

I normally don't look at Zigbee logging unless looking for something specific, so I restored the hub to 2.3.2.141 and looked at Zigbee logging in that environment. While there were some clusterId 0x19 messages, they were only from a few Iris V2 plugs and relatively infrequent. I didn't study the logs extensively and likely missed seeing some OTA traffic from other devices on the older firmware level, but it only took twenty minutes of monitoring to see that the difference between the two firmware levels in the amount of OTA cluster messages was like night and day.

On 2.3.3.125 It's happening with almost all of my Zigbee device types (Iris V2 motions and Iris V2 contacts, Iris V2 plugs, Samsung Leak Sensor, Leaksmart water sensors, Visonic MCT 340E contact sensors, Sylvania outlet, SmartThings Multisensors, GE Jasco wall switches). The only devices in my setup that aren't showing this behavior are my Sengled, Cree and Osram bulbs, Aqara and Xiaomi buttons/cubes and Iharyadi's Environment sensors.

It's not random, it's not related to noise, and (so far) my mesh isn't going crazy.

All the devices showing this behavior produce this bursty traffic every 10 minutes, like clockwork.

While I haven't noticed any performance impact on my 80-odd Zigbee device mesh, it's going to sap the batteries on sleepy devices. Not sure by how much, but I've been spoiled by multi-year battery life on most of them and don't want to give that up if this is an issue that can be corrected.

[aaiyar]

So far, I am seeing this only for those end-devices and routers for which I know Hubitat doesn't have a firmware update.

For example, I'm seeing it for my RGBGenie remotes (see below), but not my Hue motion sensors.

This is on a C-5 with the latest platform (2.3.3.125).

[jameslslate]

• I have this issue also.

[Sakman]

I dont usual look at the zigbee logging, but I believe I am seeing more traffic than usual. This is a C7 with 2.3.3.125

The arrival sensors do usual show a lot of traffic as they ping every 30 seconds or so, BUT, not the leak sensors. I don't remember the leak sensors with this much traffic.

I am also seeing this extra traffic on my 2 RGBgenie ZB-5122 remotes

My CPU load was usually less than 1% but now it is up to around 2.5%....Not much to worry about I know, but it does appear to be an increase

[erktrek]

So I haven't looked at mine in a long time either so don't know if mine is too much or what. Things seem to be working okay .

Like @aaiyar I have an RGBGenie remote but it does not seem to be that active.

I guess the worry above all else for my system is premature reduction of the battery life of my devices.

[Tony]

@aaiyar

I am seeing this only for those end-devices and routers for which I know Hubitat doesn't have a firmware update.

I'm not seeing this for my Aqara/Xiaomi buttons/cubes; I don't believe there is any firmware update for those, yet they are not producing this type of traffic. Good thing too because their wafer batteries are a pain to change.. The only Zigbee devices I have with poor battery life (6 months, typical) are the SmartThings multisensors. Unfortunately, they are part of group that now shows this periodic chatty behavior.

[kkossev]

EDIT: I have deleted my previous post, as I am not 100% sure how it was before.

This traffic on cluster 0x19 seems unusual to me, but it does not create issues with my Zigbee network. I have few devices which generate 100 times greater traffic than all these OTA upgrade requests. I haven't paid attention how it was before.

Summary

I can confirm the pattern "....five profileID 0x104 clusterId 0x19 messages .03 seconds apart, always following a profileID 0x0 clusterId 0x6 message" :

Water Leak Sensor (SAMSUNG) 0x019

Philips Hue Dimmer 0x019

......................
Button Philips Hue Dimmer Livingroom2022-10-02 00:01:03.634 profileId:0x104, clusterId:0x19, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-76
Button Philips Hue Dimmer Livingroom2022-10-02 00:00:53.631 profileId:0x104, clusterId:0x19, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-76
Button Philips Hue Dimmer Livingroom2022-10-02 00:00:43.622 profileId:0x104, clusterId:0x19, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-76
Button Philips Hue Dimmer Livingroom2022-10-02 00:00:33.757 profileId:0x104, clusterId:0x19, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-76
Button Philips Hue Dimmer Livingroom2022-10-02 00:00:28.135 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-76

[aaiyar]

@kkossev

This traffic on cluster 0x19 seems unusual to me

Maybe it becomes an issue based on its frequency. Mine is not too bad, but maybe if it was much more frequent it becomes an issue?

[Ken_Fraleigh]

@kkossev

five profileID 0x104 clusterId 0x19 messages .03 seconds apart, always following a profileID 0x0 clusterId 0x6 message

Iris contact sensors, Iris motion sensors, almost all of my Samsung devices (buttons, contact sensors, plugs, leak sensors), GE dimmers and switches. Is this the fix for “Zigbee firmware update stuck at 80%”?

[mike.maxwell]

@Ken_Fraleigh

Is this the fix for “Zigbee firmware update stuck at 80%”?

No, for sure there is no relationship to that fix.

[Ken_Fraleigh]

Any ideas? I’m just wondering what could be causing this and how it will impact battery life. I’m assuming negatively long term.

[Tony]

Btw, logging pages can be filtered by the device (by clicking on its name at the top of the page),.. makes the periodic nature very clear for what would be normally quiet devices.

I don't see any performance impacts, but I'm not measuring anything. Battery life has to get worse and the extra RF congestion won't help either. Minimal impacts, but enough to make me back off to the 2.3.2 (I just put in a fresh set of batteries on most of my sensors, and my OCD made me do it). Missing all the really neat UI logging enhancements of 2.3.3 though!!

[kkossev]

I have restored my Zniffer setup, let me know if WireShark CAP files can be of any help...

What I see is that the devices are sending Command: Query Next Image Request (0x01), then HE just acknowledges the command, then the device sends the next ZCL OTA: Query Next Image Request , increasing the sequence number, again only acknowledge from HE, repeated 5 times until the next cycle of Match descriptor request broadcast from the device and next 5 OTA requests.

Seems like this is the difference compared to 2.3.3.xxx :

[Tony]

FWIW, on 2.3.2 that I just fired up (I've had Zigbee logging running on it for over 10 minutes) the only OTA related thing I see is a single instance of clusterID 0x19 message, and only from a couple of my Iris V2 plugs. Not seeing any ZDO Match descriptor requests (profile 0x00/ clusterID 0x6's) at all, and not a 0x19 peep out of any other of my Zigbee device types. No device (other than the Iris V2 plugs) has generated any OTA traffic at all.

Also on 2.3.2, the frequency of the clusterID 0x19s for the V2 plugs is evidently greater than 10 minutes, otherwise all my Iris V2 plugs would have logged an instance by now (so I guess it follows that other device types may yet generate some OTA traffic, if I wait long enough).

[mike.maxwell]

If you're worried about network congestion or battery life please revert to 2.3.2 for the time being until we sort this out.

[Tony]

Just checked Zigbee logging again; on 2.3.2.141, the Iris V2 plugs are ticking off a single OTA clusterID 0x19 message at 30 minute intervals (or so it appears thus far). Nothing else, and no other Zigbee device type appears to have generated anything OTA related yet.

[kkossev]

I think the problem and the possible solution are clear - see the last screenshot here. 2.3.3. is missing the OTA query response from HE w/ Status: Ota No Image Available (0x98).

[Tony]

Bingo!

[danabw]

Cool...appears I have the same issue as others w/these reports, haven't noticed any clear impacts on Zigbee performance, though I did have one Iris v3 motion sensor that stopped reporting last night, I had to pull battery and restart and then it was fine. Everything else seems OK.

[craigspree]

I feel validated that I wasn’t going crazy when I saw this in my zigbee logs, and that’s it’s reproducing for people far smarter than me. I’m confident this will get fixed soon.