[Solved] Zigbee Instability is back!

Yes, we were shocked. The did confess to messing up a batch of sticks. I suspect they let their guard down somehow at their China facility -- actually I don't know how it happened. We also noticed that before about 1 in 10 boxes had a little green dot on them, suggesting those were tested units. The more recent batch had green dots on pretty much every box. As I said, we just decided we had to test every stick, and that's part of our production process now.

1 Like

Agreed. But would like to add that nobody has such a big setup. That can potentially be raise some questions regarding the stack, no?!

Yes, that is what i thought at first also. But what he describes is much worse, and the elements don't add up. Now, he's down to a device count on Hubitat that we see not infrequently, and there are still unexplainable problems. Obviously, I don't know what is causing these problems. The fact that it is quite anomalous is an important clue.

2 Likes

On the return, it was an Amazon purchase...

The bad view comes from the absoutely incredulous amount of time and money I've put into this migration. The family, who's used to pretty much full automation, is struggling with what few automations I have built not working most of the time, dealing with light switches (and forgetting to turn them off). One evening in my shoes here, and you'd probably be calling a divorce attorney.

If you want to send the gear, I'll be happy to work up one of my test plans.. QA testing is one major aspect of my career so I am willing to continue this testing for a bit longer.

I also neglected one item on the purchase list. 4 Hue outdoor floods, and 2 A19's for indoors. Those were to remove the last of the LIghtify lights which were flagged as being a potential problem early on.

Good that you have that process in place.
I was a lean six sigma black belt (havent run projects for a long while). That amount of errors is insane.

Not sure if you ever went to the factory during one of your own production runs, if not might be worth it.

Don't worry about Amazon. If you end up wanting to return it, we will refund your purchase independently of Amazon.

I understand about the incredible amount of effort you have expended. I have no doubt that you are sincere in your approach to this, and I hope you can see that we are as well. It pains me to see what you've gone through to no avail. Please send me your shipping address and email by PM, and it will go out in tomorrow's mail -- you should have it Monday.

9 Likes

Great attitude Bruce. That makes a difference

so true

PM Sent. Thank you Bruce. I hope by building a parallel mesh we an track down exactly what is going on here.

I completely agree.

2 Likes

Hopefully the new stick/hub will fix your issues. I don't have as many zigbee devices as you but I do have around 130 zigbee devices on HE and only have low battery issue. I also noticed most of my iris motion sensors blinking with blue light after a battery change and discovery is needed to get them back on HE.
I have 4 Iris contact sensors and they will show opened after a hub reboot. An open/close the window bring them back to normal state.

From my test they are the real deal.

My house is 3 story 2500 sq ft.

My hubitat hub is sitting in one corner of my basement where I keep all my hubs and equipment. I have 3 ikea outlets on the main floor and one upstairs and also have my xbee upstairs in my office.

I have about 7 leak sensors 5 vibration sensors, 7 motion sensors, 10 temp sensors and a wack of contact sensors on every door/window. I also have a contact sensor inside my metal mailbox outside, a sensor on the side door of the garage and two vibration sensors on the garage doors at the front of the house (I'm actually shocked these stay connected as they are the furthest away from anything but they get used every day and still check in). My only other zigbee devices are 5 sengled bulbs in the basement. They are not routers.

Looking at the mesh, most of the devices connect directly to the hub or to the xbee. A few connect to the routers. I thought more would go through the routers but not as many as I though. They also jump around. I have devices in the basement sitting near the hub going through a router on the main floor. Its strange. I could pair one right next to the router and its still going directly to the hub. You can't assume they are using the router. But they do bounce around.

My wifi is on Channel 1 (its a eero mesh setup) and my hubitat hub zigbee channel is 23. I saw one of the biggest changes in reliability when I got the channels right. Higher channels though caused problems with zigbee devices. I started at 26 and moved down and it wasn't until 23 that it stabilized. I am also in a detached house. When I look at wifi networks in the area I see some but nothing over powering. I only have 1 neighbor and their wifi is very basic. So my wifi interference is very minimal if anything. I could see this being more of an issue in an apartment when your surrounded by wifi routers.

I have a script running (similar to the other device monitor scripts you find on the forum) that check the last activity of every xiaomi device. With my previous sylvania plugs I had it set to notify me if they didn't check in after 3 hours. Even though they missed check ins for that long they were still working though. Their checkin is just not making it to the hub. Now that I have all ikea outlets I set the script to notify me if they don't check in after two hours. I noticed that 99% of the devices would check in every hour but there would be that one that would check in at an hour and 20 minutes. It was still working though and been using 2 hours all week and I've gotten no notifications with missed checkins for these devices.

I've had the same issues on both the ST hub and the Hubitat hub. I never got it working as reliable on the ST hub but I also didn't stick around on ST long enough to work on it.

I've spent many hours working on this to get them working. Looking at mesh's, testing routers digging through driver code and writing my own apps to monitor. Its was a big investment but it has paid off and is now stable and working great.

I may be an exception here. I don't know if anybody monitors it like I do or has as many but I keep a really close eye on it all because if they get stuck it affects other automation's such as my heating, garage doors, security system etc.

I'm starting to wonder if some of the problems too are hardware related. They seem really extreme in some cases. You just never know.

(Another long ■■■ post)

2 Likes

That is not right. You should not have to put the hub back into inclusion to reconnect these devices. I believe you've helped me confirm this as a bug.

This particular Iris motion sensor, I wound up deleting completely from Hubitat and attempted to pair it as a new device. It's been initializing for over an hour. It is less than 10" from the hub, and a foot away from one of my connected XBees. You can see the hub's stick behind the monitor. The mounted to the wall.

Hubitat absolutely will not allow this motion sensor to reconnect. Iris did in about 11 seconds. I was hoping that bouncing it to Iris and back would change the behavior but it did not.

@mike.maxwell @chuck.schwer there are now two confirmed cases of the Iris/Centralite sensors dropping off of the network when the batteries are removed.

You're in good company. I monitored my SmartThings system like a hawk. Once a week I would check every sensor to make sure it was reporting and the battery had not died.

1 Like

I just ran a quick test using a spare Iris 3326-L motion sensor. I was able to add it to my hub (2.0.4.117], remove it, and re-add it (required a battery pull to get it into pairing mode, but I did not use the reset button.) I did this twice and it worked as expected. No issues. I then left it paired to my hub, pulled the battery, and then reinstalled the battery. It recovered normal behavior without any issues whatsoever.

I am not saying there is not an issue, just that I cannot reproduce it.

BTW - My 3326-L Motion sensors are running the latest Iris firmware version provided by the Iris hub.

1 Like

I'm really hoping that you have a bum radio, not because you deserved one, but I have no other explanation for your zigbee issues currently.
We should probably hold off on further testing until you receive your replacement.
At least 10 of my 30 motion detectors in production are these guys, and I've not seen the issues you describe, so we have another anomaly here.

2 Likes

@srwhite Any good writeups on how to load the Zigbee 3firmware on the Xbee 3's?

I am thinking of trying them out.

Only thing I could think of for not being able to discover your device is because of the failed initialization and you now have a ghost device in the database. This actually happened to be before and the only way to add the device back is to wait for the database to get rid of this ghost device. Just guessing but I think the hub does this nightly.
The First pic was yesterday and Hanged during discovery. Was not able to add it back. Will try again later.
The second is a pic of my zigbee logging and you can see a devices with HEX number and no name. This was caused by a hanged initialized. Multiple reset and hub reboots did not get rid of it. I waited 2 days and was able to add it back.

Just to support this being an anaomoly; I have 6 battery powered devices on the second floor; 16 battery powered devices on the 1st floor. I had issues until I added 5 plugs. But...I have no issue with devices dropping offline.

*Note: All the devices are zigbee. I have 60+ zwave devices throughout my house.

Hoops everyone is enjoying the storm! Yesterday was a mix of shoveling and tearing down the temporary Iris installation and moving all of the plugs back to Hubitat. All 11 of the new SmartThings v3 plugs I connected on Friday remain online, stable, and instantly responsive. So that’s a plus.

To recap.. I set up an experiment by taking my old Iris hub out of storage and moving a final total of 31 Iris SmartPlugs from Hubitat, plus some battery powered devices that were dropping off my Hubitat mesh, including 2 motion sensors and 5 contact sensors. All of those devices had up-to-date firmware as reported by Iris, all remained responsive and connected to Iris for the 30+ hours of the experiment.

Before moving devices back to Hubitat, I double-checked to ensure that all of them were completely deleted so they would pair as a brand new device. There was a considerable difference pairing each plug. Some took 20-30 seconds, while others took 5 minuted or more. Several hub reboots were required, and at one point Zigbee died completely (went to “Initializing”) although that was fixed with a reboot.

This morning, 16 of the 31 plugs and ALL of the battery powered devices, all of which were reconnected as new were completely dead this morning. I even powered down the hub for a solid hour. I still cannot even wake them up the plugs by power cycling them, refresh, etc. I probably should have left them on Iris, but I personally utterly and completely detest their mobile app. The only common thread is that these are Centralite devices. The new SmartThings plugs (Samjin) are all working fine.

Thanks to @bravenel who is sending another hub and stick (actually 2), there’s still quite a bit more testing to go. He’s offered some test scenarios to work through. With any luck I’ll be able to start that on Tuesday. Fingers crossed that it’s just a bad stick!

4 Likes

Best of Luck mate

1 Like

Yes all the best. I think there are quite a few people keeping an eye on the issues you’re having and hoping a successful out come for you is found soon.

5 Likes