C8 Zigbee mesh is a hot mess

🎬 Get Started Using C-8
60

A a part of your troubleshooting process, check your 2.4GHz Wi-Fi channels (yours and your neighbors could interfere) against your Zigbee channel setting (on Zigbee Details page). See my post above w/the Wi-Fi/Zigbee channel comparison.

Before you reset try doing a local backup of your hub database, and then simply restore the backup you just downloaded. The database will be "cleaned" as a part of the download/restore process and could help. After the restore, shut down your hub and remove power for a minute and then restore power and let it boot up. See if things improve...

4 Likes
61

Hi, thank you for your guidance. I just found the time to do that today and am crossing my fingers that it's going to work without me having to exercise the nuclear option. Here's hoping. (^_^)x

1 Like
62

I think you're referring to doing the local backup/restore, correct? That's good.

Also do the review of your Zigbee channel set on your hub vs. local and neighbor 2.4GHz Wi-Fi channels to ensure you don't have 2.4GHz Zigbee sitting on top of 2.4 GHz Wi-Fi. A good tool for that on a phone is the Unifi WifiMan app.

2 Likes
63

Hello. Thank you for all that information. I was struggling to find the Wifi channels that my routers use (one would have thought they would be on the app, but no), so I downloaded the app you suggested and found that my home wifi network pretty much spans the 2 through 10 channels. I'm changing my Zigbee channel to 25 and will see what happens.

With regard to your clarifying question: yes, I downloaded a local backup to my hub, and then restored it from there. It might have changed the behaviour but I'll have to reassess.

Thanks again for walking me through this.

1 Like
64

Think I'm in a similar boat here with my Zigbee performance. I've been lately battling the issues of devices not wanting to connect, or stay connected. I tried a few weeks back restoring a backup, which might have helped initially, but things have still been acting up.

I have a couple Sengled non-repeater plugs (the older square ones E1C-NB6) that don't seem to stay connected sometimes for even an hour. These have been running my Christmas lights, so I haven't really noticed the issue until now.

Even more troubling, is my Ikea Tredansen blinds. I have 3 in my master, and maybe around a month ago one stopped responding. Initially I thought it was a dead battery, so I ended up pulling them and charging them. Then, none of the 3 wanted to respond. I was able to factory reset and repair, but even after the hub picking them up they would not respond to commands.
I ended up removing one of them from the hub, factory reset and re-add, and now I cannot get the device to pair to the hub. For info, I am getting this screen in the pairing menu, but then it seems to timeout and the initializing note just goes away.

Zb pairing
Zb pairing1407×768 58.8 KB

Tried standard pairing, as well as the other two options with the same results. I even grabbed one of the Ikea repeaters that came with the blinds, but couldn't even get that to pair to the hub in the same room, less than 8 feet away.

Zigbee hub power was on 8, but after some reading tonight I cranked it up to 20. I'm going to let things sit a day, and maybe even restore from backup and see how it goes.

Sometimes the Smartthings motion sensor in my master bath stops responding, but I have found pulling the battery and putting it back in gets it working again for a while.

Any tips or recommendations would be appreciated! I've thought about resurrecting the C7, and setting it up as a secondary hub to see if that helps, but really would like to avoid that if possible. If my list is up to date I have 60 Zigbee devices, and 32 are mains devices. I have plugs, mostly Sengled E1C-NB7 or Sylvania Smart+ (11 devices in total), 1-2 in pretty much every room. The other mains devices are Gledopto RGB+CCT controllers for LED strip lighting (7 total), or Ecosmart bulbs (6 Total).

I am hoping things will stabilize and stay connected, but it seems most things I try work for a few days or a week, and then start acting up again. I've even tried moving the hub to different locations, but get pretty similar results without monitoring too heavily. Unfortunately, without being able to plug into a POE adapter, I am fairly limited where I can put it due to UPS availability. Either in my basement, or on my second story, neither of which are optimal. With the C7, I had it on the main floor, though it was in the corner of the house in this location, it never had these issues. The second story location is almost exactly central to the house, though it still tends to be more problematic than the C7 in the corner of the main floor. The basement, seemed to perform the worst, for obvious reasons of being furthest from most devices.

65

C-8 is the only HE hub to use an SoC based on the Zigbee 3.0 protocol; while this sounds like an upgrade (there's literally no performance benefit) for the typical HE user with legacy Zigbee HA profile devices, there's a significant downside: devices that disconnect might be unable to transparently rejoin the mesh as they normally would if joined to a non Zigbee 3.0 hub.

It's not disconnects that are the issue; they happen even in a properly functioning mesh for a variety of reasons: routine battery replacement, temporarly signal blockage, momentary power interruption to their parent router... Zigbee devices are designed to recognize when their link is lost and go through a rejoin process to reconnect. We expect this to happen without user intervention; that's the experience most of us have with C-3 through C-7 hubs (at least once the mesh is stable).

The improvement Zigbee 3.0 provides-- at least in a mesh consisting of all 3.0 devices (along with multiple profile support, irrelevant to legacy Zigbee users)-- is fixing security vulnerabilites caused by use of a default link key (the link key is used during joins/rejoins to encrypt transfer of a unique network key used for subsequent communication). Legacy devices all use the same 'well known' link key during initial join and insecure rejoins to obtain the unique netkwork key. 'Pure' Zigbee 3.0 networks fix that problem but create new issues when legacy and 3.0 devices must interoperate.

The Zigbee 3.0 security model doesn't use a default link key for device joins-- instead a unique link key can be embedded in the 3.0 device (printed on the device itself and entered 'out of band' when the device is initially joined) . If the C-8 adhered to this security model, it would then be incompatible with older HA devices which can only join use the default link key.

Unfortunately meshes that must use legacy devices with a Zigbee 3.0 coordinator face issues-- documented by Si Labs in one of their app notes Silicon Labs Community :

My guess is that the C-8 takes this approach:

"Using the default link key as a transient link key (a link key with a timeout after which it will no longer work). This allows HA devices to join a network for a small window, minimizing security vulnerabilities, but it can cause problem if a legacy device associated using the default link key detaches from the network, it won't be able to rejoin using an unsecured rejoin"

Per Silicon Labs, to get the same level of reliability (that is, rejoins without user intervention) that we've been used to, but still using the C-8 with legacy devices, would require:

" Using the default link key as the primary link key within the network. This solves any HA interoperability issues, but with the concerns of wide open security".

Personally I'd be fine with that as a configuration option; fortunately the only one motivated to hack my Zigbee network is me. I have a C-8 but it's been unused; frankly it would be my last choice (after my also yet to be deployed C-7; flaky Z-Wave kept that one on the shelf) to replace my nearly 100% rock solid C-3 as Zigbee coordinator.

9 Likes
66

I have 105 zigbee devices attached to my C-8. using channel 25. I have no complaints. When I first got the migration done from C-5 to C-8 there were some strange things happening. Including dropping off the mesh. However, the devices came back by themselves a few hours later. Been very happy so far.

1 Like
67

Thinks were wonky with c8 (in terms of dropoffs) till i got sonoff 3.0. Dongles which makes sense from reading above... however initial joins can still be a whore/chore.

1 Like
68

Thanks for the descriptive post! Makes a lot of sense as to what's going on in the background. I believe this is what is happening in my situation.

I personally would love the option to reduce security to see if it solves my problems, even as a temporary test.

Still no luck getting my Ikea blinds to pair. I even took one of them down and brought it into the same room as the hub and tried pairing. Similar issue as before, where it finds a Zigbee device but doesn't get past the initializing stage.

Zigbee is currently on channel 20, and my WIFI is on channel 11. Channel 11 is optimal in my location with other WIFI networks nearby. Though looking at the channel graphs there might be some channel overlap causing issues. I have 2 AP's, one is maybe 10-12 feet from the hub, through a wall though. I think my next step is to play with WIFI/Zigbee channels to see if that will help my performance at all. Think I am going to switch Zigbee to channel 25, and let it sit for a few days. If that doesn't seem to make much of a difference I will swap my wifi channel to 1 or 6, and see if that helps. Hopefully things work out, but time will tell

1 Like
69

For reference, do you know what channel your WIFI is set to?

70

Well positive update, but still monitoring things. I did some more WIFI analysis, and ended up switching my WIFI channels to 1 for my home networks. Seems to be a lot of traffic on channel 1, but most of it had really low signal strength, so I'm not too worried about interference. Having 2 AP's definitely helps with range and signal issues as well, from a WIFI standpoint.

Kept my Zigbee channel on 20, with power still set to 20. Things seem to be responding well, and the outlets I was having problems with are all connected and responding. They all came on today when they were supposed to, which is the first time recently they are all behaving.

I did add a mmWave sensor to the mix today, and didn't have any issues pairing it in the same room as the hub. Moved it a few rooms over, and it seems to be working nicely.
Still haven't messed with the Ikea blinds today, going to wait a little more before I do that. Also, its just been me home today, so having a few less WIFI devices here is probably a nice bonus.

If all stays cooperating, I'll probably start dialing back the Zigbee power level, as to reduce strain on the Hub's radio. So far, pretty happy, but I'm going to keep monitoring to see if it keeps up

1 Like
71

When you get to that, if everything else fails, try keeping the blinds (the end where the battery is) really close to the hub, to make sure they pick the hub as their parent (and not any other zigbee router).

I don't have the blinds, but this procedure is required to pair the other battery-powered IKEA devices (buttons).

1 Like
72

Thanks for the tip. In the meantime I repaired the Ikea buttons to the blinds so I can use them without walking up to each one and hitting the up or down button (I know how terrible). In their instructions they mentioned holding the button really close to get them to pair. I'll probably pull them all down and pair them right next to the hub and see if I can get it working again. Might try it tomorrow, if everything else is maintaining.

73

What I find most stable for most people is 6 and under for wifi, 25 and above for zigbee with 12 for power. For myself and helping others, it seems to cure 99% of the issues...(caveat... issues that don't involve aqara, xiaomi, and sonoff)

74

@Tony
Thanks for that very clear and insightful description of Zigbee 3.0.

Is that the reason for the pairing option "avoid Zigbee 3.0 repeaters"? '
And, is that how they do it? (speculative, of course.)

75

Thanks, WIFI is currently on channel 1, but Zigbee is still on 20. Thinking I am going to need to try changing Zigbee to channel 25 to see if that helps. Saturday and Sunday, everything performed pretty well, but yesterday (Monday) I noticed the two older Sengled outlets (E1C-NB6) I have been having issues with are still acting up.

I was able to get an Ikea repeater plug to pair, and stuck that in the room with the two problem devices. Going to try to get them back online today at some point and maybe since the repeater is there they will start cooperating a little better. For whatever reason that room seems like sort of a dead zone for these plugs. Its funny though, because I have two of the newer Sengled plugs (E1C-NB7) in the same room, and they both are perfectly fine. I have 4 total of the older style Sengled (E1C-NB6), and the two that are in different locations are totally fine, but the two in this room aren't.

Honestly, since you mention Aqara, it might be related to these devices. I have 5 temp & humidity sensors, and 2 contact sensors (all battery devices). I noticed a couple had fallen off and re-paired them Sunday evening. Seems like once I did that then the other things began acting up again. Previously things seemed to play pretty well with the Aqara devices, aside from them randomly dropping off mesh and needing to be reconnected.

Any thoughts what might be best to try next? Remove Aqara, try channel 25, pull the old Sengled outlets and light them on fire? :joy: I do like the older Sengled ones because they aren't repeaters. I have been using them for holiday lighting and temp test switches mostly. So if they get pulled and thrown in a drawer for a little while in the past it really hasn't bothered my mesh. Though it seems the Zigbee gods are punishing me for it, because I know this isn't the best practice :zap:

76

Non-repeating Zigbee devices should not be impacting your mesh unless they are bombarding it w/excessive activity and/or have a very noisey driver, which I doubt they are/do.

Aqara devices in general are unlikely to hurt your mesh more than any other device, unless similar issues w/noisey drivers that are generating excessive traffic or errors due to calls to deprecated features. I run Aqara on my C7 & C8 and do not have issues w/them causing problems. On my C8 they have actually been more reliable than Iris v2 motions and Visonic contacts.

Options for Aqara drivers

I'm mostly using this one for my Aqara contacts.

What settled my C8 down the most seems to be repeaters taking the load off the hub...in recent testing I've determined that pretty much all of my Zigbee battery devices on my C8 are routing through repeaters rather than directly connected to the hub. My best repeaters are SonOff USB dongles, Tuya USB repeaters (the tiny ones), Iris plugs, & Ikea plugs. C8 power is set to 8.

During the time when devices were repeatedly failing to stay connectd on my C8 (and oddly, this was led by usually very solid Samsung/Centralite leak sensors, Iris v2/v3 motion sensors, and Visonic contacts) I moved those devices back to my C7 and hub meshed them back to the C8. That seemed to be the other key part of things settling down and I've been running pretty much issue-free since then. ~90 Zigbee on C8, ~35 Zigbee on C7. Stock Zibee antenna on the C8.

I did do an antenna mod for Z-Wave on the C8 - it sorted some issues I'd had forever on the C7 and to less extent on the C8 w/some Z-Wave iBlinds that had trouble communicating once in a while to the hub. The antenna (Bingfu 4G LTE RP SMA Cellular Antenna 7dBi Outdoor Antenna) provided solid connectivity (the iBlinds are communicating through the metal housing of my blinds) and they have been perfect since.

5 Likes
77

I'm not sure how this option would be able to address the link key issue (or even if it was intended to). Zigbee uses a 'trust center'-- a logical entity responsible for distributing security keys. Zigbee doesn't require the TC to reside in the coordinator, but that's the way it works in Hubitat's implementation: any router intercepting a join request acts as intermediary and passes requests for keys to the trust center so (in a Hubitat mesh) all security key distribution is handled by the hub.

That being the case, I don't know how avoiding Zigbee 3.0 repeaters during pairing would make it easier for legacy devices to join.

AFAIK , a joining end device simply responds to one of several broadcast 'beacons' from any router in range that has child capacity to accept the join. I don't know (there's a LOT I don't know) how a joining device could distinguish a 3.0 repeater's beacon; maybe that distinction is made at the hub when the key request gets passed to the trust center (I assume that the hub can distinguish the protocol version of any router that it is communicating with; maybe the hub selectively issues a 'network leave' request if the joining device's link is via 3.0 repeater).

It's a mystery... just like 'network rebuild'. I can see that function's affect on the hub's child table (it seems to purge it of entries that don't exist anymore) but whether or not it has any affect at all on the rest of the mesh is unclear (my uninformed guess is that it doesn't).

1 Like
78

Thanks for the Aqara notes! I've been following your progress with Zigbee instability for a little while now. Your stability of bringing the C7 back and using both hubs made me think about doing the same, but I wanted to try and see if I can get things to cooperate a little better before taking the more drastic approach of running two hubs.

I've been using this driver for the contact sensors, this driver for temp sensors, and this for my leak sensor. Forgot to mention I have an Aqara leak sensor too. The devices keep piling up, I forgot what I have around the house :joy:

Researching in the thread you linked with Aqara driver recommendations, someone mentioned the drivers by Marcus could be problematic. Turns out I have my contact sensors running those drivers, I recognize the warn logs from the screenshots.

Was able to get all the contact, temp, and leak sensors swapped over to the chirpy driver, and they seem to be responding well initially. I did initially swap to the 'device' driver, clear all states/schedules/etc, and then swapped to the new driver. Some of the temp sensors liked dropping off after a while, so hopefully the driver change will help with that. Fortunately, they don't do much other than log temps for reference, so no real harm if they do drop off.

As for the older Sengled plugs, I am thinking I might have to replace them with different devices. Early this evening, the two cooperating ones turned on when intended. Though after some testing tonight, neither wanted to turn off when triggered from a dashboard or the webpage. The two problematic ones I cannot get to 'pair in place' where they are located. I can only get them to pair when they are in the same room as the hub (I have been pairing them within feet of it). I have 6 of the newer Sengled NB7 sitting in boxes that I picked up on Black Friday, so I think I am just going to unpair these, and swap them out with the newer models. Less hassle, and I want lights turning on and off properly when I am not home. I'll probably save these ones for a rainy day, or when I need an outlet to do some quick testing with.

I also ordered one of the Sonoff USB dongles, should have it tomorrow. Though I don't know if I will get a chance to play with it before the holidays. Figured it may be a better solution than the Ikea repeater in my dead zone room, to hopefully help devices stay connected.

Any good ways to tell if a device is using a repeater or not? I know the Zigbee graph shows this with the lines, but it can get pretty messy to read sometimes. Didn't know if there was a more effective way to track repeater usage.

79

Just have a moment to reply, more tomorrow, but wanted to say that running two hubs really is no biggie, and was a core part of my solution. I waited way too long to start moving some of my devices back to my C7... So don't be too shy about trying that. :+1::slightly_smiling_face:

3 Likes