Zigbee Mesh Strength

[kerrylebel]

Hi All,

I have over 65 GE Zigbee switches and dimmers in my house across two floors. Below is a segment of my zigbee logs. Anyone have any idea why my switches are constantly not working? My Wifi is on Ch 1 and Zigbee is on 25. I have shutdown the hub for 30 minutes, I have factory reset devices and readded them, I have pulled out all endpoint devices, etc. Nothing seems to change it. I’m considering picking up a cobee zigbee usb and using Wireshark with it. Anyone ever go this route to troubleshoot?

Master Bedroom Bulb 32021-12-20 21:46:16.122 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Guest Hallway Motion Sensor2021-12-20 21:45:59.649 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Guest Hallway Motion Sensor2021-12-20 21:45:45.371 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Guest Hallway Motion Sensor2021-12-20 21:45:39.742 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Guest Hallway Motion Sensor2021-12-20 21:45:29.728 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Master Bedroom Hallway Lights2021-12-20 21:44:55.056 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-63

Guest Hallway Motion Sensor2021-12-20 21:44:53.043 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Master Bathroom Motion Sensor2021-12-20 21:44:48.620 profileId:0x104, clusterId:0x19, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-56

Guest Hallway Lights2021-12-20 21:44:44.902 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-64

Guest Hallway Lights2021-12-20 21:44:44.899 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:200, lastHopRssi:-60

Guest Hallway Motion Sensor2021-12-20 21:44:43.085 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Master Bedroom Bulb 2 2021-12-20 21:44:37.853 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Master Bedroom Bulb 12021-12-20 21:43:13.013 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Guest Hallway Motion Sensor2021-12-20 21:38:57.804 profileId:0x104, clusterId:0x400, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Master Bathroom Motion Sensor2021-12-20 21:38:22.601 profileId:0x104, clusterId:0x400, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-56

Master Bedroom Bulb 32021-12-20 21:36:26.846 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Guest Hallway Lights2021-12-20 21:35:01.007 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-65

Guest Hallway Lights2021-12-20 21:35:01.000 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-64

Guest Hallway Lights2021-12-20 21:34:59.471 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-64

Master Bedroom Hallway Lights2021-12-20 21:34:46.988 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-49

Master Bedroom Bulb 2 2021-12-20 21:34:44.069 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Master Bedroom Bulb 12021-12-20 21:33:18.953 profileId:0x0, clusterId:0x6, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-65

Guest Hallway Motion Sensor2021-12-20 21:32:54.495 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-63

Guest Hallway Motion Sensor2021-12-20 21:32:40.109 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-62

Guest Hallway Motion Sensor2021-12-20 21:31:58.350 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-63

Dining Room Wall Lights2021-12-20 21:31:43.750 profileId:0x104, clusterId:0x8, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Dining Room Wall Lights2021-12-20 21:31:43.733 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Guest Hallway Motion Sensor2021-12-20 21:31:40.003 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-66

Guest Hallway Motion Sensor2021-12-20 21:31:37.195 profileId:0x0, clusterId:0x13, sourceEndpoint:0, destinationEndpoint:0 , groupId:0, lastHopLqi:255, lastHopRssi:-41

Guest Hallway Motion Sensor2021-12-20 21:31:35.574 profileId:0x104, clusterId:0x406, sourceEndpoint:2, destinationEndpoint:255 , groupId:0, lastHopLqi:255, lastHopRssi:-47

Guest Hallway Lights2021-12-20 21:31:24.000 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-59

Guest Hallway Lights2021-12-20 21:31:23.996 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-59

Guest Hallway Lights2021-12-20 21:31:22.483 profileId:0x104, clusterId:0x6, sourceEndpoint:1, destinationEndpoint:1 , groupId:0, lastHopLqi:255, lastHopRssi:-65

[ogiewon]

Looks Like you have some Zigbee bulbs in the mix. What brand/model are these?

There are definitely some known “bad repeater” Zigbee devices. By removing these from the mesh, often stability and performance is greatly improved.

Also, chatty power reporting devices can overwhelm a mesh network.

So, more information regarding your network devices would be very helpful.

[rcjordan]

Based on repeated recommendations here, I switched to channel 20. (C4, large house, 2 floors)

Zigbee and Zwave Channels

Zigbee channel 20 seems to be the most common used and recommended channel. Reason is it sits between Wi-Fi 2.4 channels so less conflicts. I’d start there and if you encounter issues you can always change it. Stay away from 26 because some Zigbee sensors don’t work with it and it’s also a weaker channel.

[Tony]

You said your switches are constantly not working; does that mean the sensors are working? Are there any devices (or subset of switches) that work consistently?

If all Zigbee is hosed and nothing works reliably (and you've ruled out the hub's Zigbee radio), there may be some serious RF environmental issue or signal blockage. But if only a subset of your devices are problematic, they might be sharing a failing repeater or weak link in the mesh. Without a mesh map you don't get an indication of where the choke point may be.

Note also that Zigbee logs only show an indication of the last hop. Meaning, if they are not from child devices or neighbors, the RSSI figure you're seeing is that of the last repeater they used, and then, the log entry only appears for a message that actually got through. Completely failed 'first hops' in a multi-hop route won't even result in a log entry.

To put it another way, the RSSI's all look fine. But what you're not seeing is any indication of the signal strength on the first hop the device took (or through intermediate repeaters on multi-hop routes). So a weak signal (or link subject to interference) may get retransmitted and logged with the 'good' RSSI values you're seeing for that last hop.

Lacking Xbee/XCTU mesh mapping, one thing you can do when you have several devices consistently showing the same last hop RSSI's is make a few (possibly inaccurate) assumptions. For example, there's a good chance that all the sensor devices showing last hop RSSI of -41 all share the same last hop repeater (likewise the ones all showing -66). So if you trigger a mains-powered switch that is a neighbor of the hub (that is key) and it generates a log that shows RSSI of -41, it's likely to be the last hop repeater for those sensor devices showing -41 as well. No guarantee, but its the best guess you can make without an actual mesh map.

How's the neighbor table look (from getChildandRouteInfo). Is it stable?

[kerrylebel]

Ah yes the bulbs. They are Sengled.

[kerrylebel]

@Tony

You said your switches are constantly not working; does that mean the sensors are working? Are there any devices (or subset of switches) that work consistently?

If all Zigbee is hosed and nothing works reliably (and you've ruled out the hub's Zigbee radio), there may be some serious RF environmental issue or signal blockage. But if only a subset of your devices are problematic, they might be sharing a failing repeater or weak link in the mesh. Without a mesh map you don't get an indication of where the choke point may be.

Note also that Zigbee logs only show an indication of the last hop. Meaning, if they are not from child devices or neighbors, the RSSI figure you're seeing is that of the last repeater they used, and then, the log entry only appears for a message that actually got through. Completely failed 'first hops' in a multi-hop route won't even result in a log entry.

To put it another way, the RSSI's all look fine. But what you're not seeing is any indication of the signal strength on the first hop the device took (or through intermediate repeaters on multi-hop routes). So a weak signal (or link subject to interference) may get retransmitted and logged with the 'good' RSSI values you're seeing for that last hop.

Lacking Xbee/XCTU mesh mapping, one thing you can do when you have several devices consistently showing the same last hop RSSI's is make a few (possibly inaccurate) assumptions. For example, there's a good chance that all the sensor devices showing last hop RSSI of -41 all share the same last hop repeater (likewise the ones all showing -66). So if you trigger a mains-powered switch that is a neighbor of the hub (that is key) and it generates a log that shows RSSI of -41, it's likely to be the last hop repeater for those sensor devices showing -41 as well. No guarantee, but its the best guess you can make without an actual mesh map.

How's the neighbor table look (from getChildandRouteInfo). Is it stable?

Thank you Tony you are heading exactly in the direction that I am. That is why I was asking if anyone used a zigbee stick and wireshark. I was aware that the logs were only showing me last hop. I have zero experience using wireshark so I am considering that a last resort to help identify where the issue is before a replace thousands of dollars in switches. As for the table, its pure chaos and always has been. Ive never understood how they setup the routing. I have switches that are 15 ft away from the hub that are routing through a switch in my basement. 90% of what shows up in that table looks more to me like the LEAST efficient route, nowhere near truly effecient and never gets better after a 30 minute shutdown of the hub. Also to answer someone else’s question. None of the zigbee devices work consistent. I only mentioned the switches as they were most important.

[ogiewon]

The Sengled bulbs are designed to NOT be Zigbee repeaters. Thus, they are not a problem.

Must be something else

[Ken_Fraleigh]

@kerrylebel , there was someone on here around a year ago (maybe longer) with a similar complaint. I believe they also had around 60 GE Zigbee dimmers and had nothing but problems. I had problems with my Zigbee mesh when I had over 60 Zigbee bulbs (even though I had only bulbs on it). I saw the problems completely vanish when I got that number down below 40. It might be the same issue you are having, might not. I saw constantly changing routing tables on my HE with all of the bulbs, and route tables that never seemed to change on my HE with just solid repeaters and end devices. The way I brought my number of bulbs down was to buy a bunch of Hue lights and use a couple Hue bridges. Since you are using GE Zigbee dimmers, you could just add another HE hub to offload half of the dimmers, assuming you don't have power reporting enabled on a bunch of them, Adaptive Lighting, or some other easy fix. For me, the extra expense was nothing compared to all of the time I was wasting trying to get things to work properly.

[kerrylebel]

@Ken_Fraleigh

you could just add another HE hub to offload half of the dimmers, assuming you don't have power reporting enabled on a bunch of them, Adaptive Lighting, or some other easy fix.

Thats a good idea to try. I actually have a spare C7. A few months back I got desperate and started thinking my hub was the issue, so I bought a second one and started completely from scratch. I will dust off the original hub and add it in and see what happens. I'll try anything at this point.

[ogiewon]

If you go down the two Hubitat Hub Zigbee Networks route, a few things to consider.

1. Make sure each hub's Zigbee radio is using a different channel. I recommend only using Zigbee channels 15, 20, and 25, as those are least susceptible to 2.4GHz WiFi interference. My personal choice is channel 20, as it has worked the best with all of my Zigbee devices. Note: Zigbee Channel 26 does not operate at full power, due to FCC restrictions. This is why I do not recommend it. Also, some older Zigbee devices will not successfully join a network that uses higher number channels.
2. Make sure ALL of your WiFi Routers, Wireless Access Points, Range Extenders, etc... are only using 2.4GHz channels 1, 6, and/or 11. Also make sure that the 2.4GHz radios are only allowed to use 20MHz channel width. This will ensure your WiFi 2.4GHz does not interfere with your Zigbee networks.

3. You mentioned that you have Sengled Zigbee bulbs. Please be sure to leave enough device that support Zigbee Repeater functionality paired to the same hub at the Sengled bulbs. The Sengled bulbs are not Zigbee repeaters, and thus will need to rely on other devices to create a solid Zigbee mesh network. Strategically divide up the Zigbee repeating devices between the two networks so both have a strong mesh, each on their own Zigbee channel.
4. If any of your Zigbee devices support power/energy monitoring, I suggest completely disabling that functionality for now. Too many devices constantly reporting minor changes in power usage can quickly create chaos on a mesh network. In order to get things to be stable, please disable this feature on any switches, dimmers, or outlets.
5. If you have any Peanut Zigbee outlets in your mesh, I would recommend removing them for now, to see if things calm down. Peanut outlets have been know to cause Zigbee networks to constantly rebuild routes between devices, causing performance issues. Cree Zigbee bulbs, and older Sylvania Zigbee bulbs, are also known 'problem devices' that can wreak havoc on a Zigbee mesh network.

[Ken_Fraleigh]

@kerrylebel

I will dust off the original hub and add it in

Plan how you are going to divide up your devices before you start. Upstairs and downstairs, left and right, whatever it is, make sure that you have plenty of repeaters near the hub.

[Tony]

The most important thing in establishing a Zigbee mesh is to get the hub showing a stable neighbor table. It only needs to have one stable neighbor to begin routing, but won't work properly if they're constantly getting bounced in and out of the table.

With regard to wifi interference, the many Alliance white papers and academic studies that have been done on this phenomenon show that Zigbee networks still continue to function, even in environments with packet rates and channel selection purposely selected to cause havoc. What happens in the worst cases is major packet loss, sometimes 80% or more, but they still work.... with increased latency (on the order of hundreds of milliseconds), but they remain functional. You've likely got something else going on; if I were to guess there are one or more flaky repeaters fouling things up.

Start by trying to get a couple of solid repeaters that stay in the neighbor table and show reasonable inCost/outCost (none should show zero). I wouldn't worry about getting more than that going; in fact for purposes of troubleshooting start by disabling as many of them as you reasonably can until you find a subset that works-- maybe you have a switch (perhaps more than one) with a flaky SOC that keeps restarting and never lets your mesh stabilize. Don't worry about weeding out a bad sleepy device; they aren't capable of disabling your mesh. If you can get at least one or two stable first hop repeaters, then start enabling the others till things go south again.

Also, I wouldn't try to second guess the routing. Links that show better RF performance get prioritized over those that are worse, bottom line; this happens constantly, even without doing the 30-minute heal (that technique is way overused; unlike self-destructing Z-Wave, Zigbee really is self-healing). Hubitat's antennas don't seem to perform exceptionally well, so first hop repeaters get a lot of action; in my house any Zigbee plug (even those a dozen feet from the hub) will become a preferred parent to a sleepy device adjacent to the hub instead of the hub itself. The fact that a repeater in the basement is preferred over one nearer the hub might be due to the way the antennas of the devices are oriented or some material obstruction. The extra distance isn't hurting you; even an extra hop count won't be perceptible if the messages are going through without retries.

[arnb]

@ogiewon

Note: Zigbee Channel 26 does not operate at full power, due to FCC restrictions.

Thank you for this information, I've been using 26 for years. Switched to 25, most devices followed in a few minutes.

Update: noticed a major positive change in the Zigbee getChildAndRouteInfo data. Went from 5 Child connected and 2 unknown devices, to 14 Child connected and zero unknown, and the Neighbor LQIs improved. One previously low LQI repeater device is up over 50 points. All devices now on Chan 25 including the mailbox, 50ft from its closest repeater.