I know it seems like voodoo, but I ran for a week or two at level 4 (at request of Support) and my ZigBee Off/On Events stopped, and I had no other ZigBee problems since - even after moving it back up to 8 (I know it's not logical - I used to have Off/On Events on 8). My stats weren't as good on 4 (LQI, outCost) but my devices still worked. Now I've been on 8 for quite a while with no issues at all - no dropped devices and nothing in my Hub Events log that shouldn't be there. I'm curious if you will have Off/On at level 4, and if your devices will find routes that keep them functioning.
OK...reporting back after changing Zigbee radio back to 16 yesterday. Haven't had a device fall off since the power change from 12 to 16. It's only been about 24 hours or so, but a couple of my leak sensors at different ends of the house had been falling off pretty frequently so this could be signs of improved stickiness of my devices. But still early days after the change yesterday.
No Zigbee radio reboots since yesterday morning when I was doing a bunch of pairing activities trying to re-join a couple devices that had fallen off before the power change back to 16.
While I was re-joining a couple devices that fell off when my hub was still on power level 12, I noticed an issue w/devices not responding to their normal reset commands. Works like this:
- I try to pair a device (and I've been working w/my most common devices, Visonic MCT-340 contact sensor and Iris v2 motion) on the C8
- The pairing fails
- I am then unable to get the devices to do a full reset. I follow the same steps I've been following for years to reset them:
-- Iris: Remove battery >10s, insert the battery w/the button held and solid red LED appears for a couple seconds, then blue pairing LED starts blinking
-- Visonic: Insert the battery while holding the tamper switch and solid red LED apppears for ~1s and then dims slightly, release tamper switch and red LED starts blinking
Rather than resetting properly, both devices ignore the reset actions and go back into pairing mode immediately (blinking LEDs).
Leaving the battery out longer on either device doesn't help.
What does fix this issue is pairing the devices to one of my C7s (they pair immediately/easily) and then removing them from the C7. After the pair/removal from the C7 the reset steps then work normally on both devices. I replicated this with multiple examples of each device, so not an instance of one or two bad devices.
TL;DR - It seems like a failed pairing on a my C8 leaves something behind on the device or C8 or both that blocks my ability to get the devices to do a full reset. If I pair them to a C7 and then remove them from it I can then reset the devices as I always have. If the devices go through another failed pairing on the C8 then the same situation w/them not responding to reset actions repeats.
No idea if this is a "just-me" or a "just some Zigbee 1.2 devices" issue, or a little of both. But I haven't run into something like this before.
I had my one Iris v2 motion stop reporting yesterday. I reset it and then tried pairing, but it was never found. After that, the device would not do the normal LED color cycle to show the reset happened. It also just immediately started flashing blue indicating it was in pairing mode.
I ended up pulling the battery and leaving it out overnight. It re-reset and re-paired the first try this morning.
Thanks for confirming the issue. I assume like me, you don't remember seeing this type of behavior previous to the C8, correct?
I left one of mine overnight and it still wouldn't do the full reset, so there is some variability there, it seems.
Well, I never had a ZigBee device drop on the C-7, so no.
Ah, yes, that would be true for me as far as I remember (especially in the case of the Iris v2).
I can confirm that happened with one of mine also. It had never dropped off my C-5. But once it started dropping off my C-8, I couldn't get it to fully reset. It would immediately go into pairing mode without resetting first. I pulled the battery and let it sit over night. Then it reset and repaired, and it's been OK for a couple of weeks now.

I pulled the battery and let it sit over night.
Thanks for confirming the same issue.
So now I'm thinking that I am mis-remembering what I did/how long I let the Iris v2 sit, since two of you have had it resolve when you let it sit overnight.

let it sit overnight.
It may have been longer than overnight, actually. I was thinking about throwing the thing away. I took the battery out because I had just put in a new one, and set the Iris aside to add to either the bad-actors-drawer or the trash can. I got lazy, and there it sat until I decided to give it one more try.

Thanks for confirming the issue. I assume like me, you don't remember seeing this type of behavior previous to the C8, correct?
You give the hub more credit than it deserves A device should be able to reset itself even if actively connected to a controller, regardless of what the controller does. If the device isn't resetting properly, that's a device problem.

changing Zigbee radio back to 16 yesterday
Maybe 20 would be better yet. Come to the dark side

Maybe 20 would be better yet. Come to the dark side
I've been considering this, you have said a few times that your Zigbee is doing ok.
If you need to set your power to 20, you might be better off adding some repeaters to the mesh. Zigbee is designed to accommodate connection through a network of repeaters rather than a hub and spoke arrangement.

-- Iris: Remove battery >10s, insert the battery w/the button held and solid red LED appears for a couple seconds, then blue pairing LED starts blinking
Just happened to review the Iris V2 pairing videos (there are a couple online for both motion and contact) and noticed a step I hadn't remembered: the video says 'hold for 2 seconds' referring to the reset button after inserting the battery.. Is it possible you're releasing the button coincident with the battery insertion? Have to admit I've reset these things dozens of times and just look for blinking blue LED as a signal that reset was accomplished.
Aside from a partial/failed reset sequence, agree with @bobbyd that there's no way a hub can leave traces behind that would survive a factory reset on a properly functioning device.

Zigbee is doing ok
It has been solid as a rock.

adding some repeaters to the mesh
In my case, I have no child devices and 16 nearest neighbors all with incost and outcost of 1.

Is it possible you're releasing the button coincident with the battery insertion?
Thanks for asking, good question. Yes, I'm aware of the 2s requirement...I actually re-reviewed the reset instructions repeatedly after the issue began showing up and I couldn't get my devices to reset. "WTF - am I doing something wrong?!"
The flows that I see w/the device when I do a reset:
- Normal: The red LED shows pretty much as soon as the battery is inserted while you're holding the button on the side of the sensor. Release after 2s and it goes into blue-blinky pairing mode.
- After failed pairing: The red LED never appears, no matter how long you hold the button (1s, 2s, 3s, 4s, etc. The LED starts blinking blue for pairing mode, but really isn't in a good state to join the C8 and will never pair for me.

there's no way a hub can leave traces behind that would survive a factory reset
That's actually my point...after a failed pairing it is impossible to factory reset the sensor, it's in some odd state where it won't reset. The device just returns directly to pairing mode (though it never joins after it's in this state) and won't reset unless I join it to my C7 and then remove it, or as @anon47916022 and @jabecker noted in their cases, leave the batteries out for many hours (overnight or more).
That does seem to indicate to me that the device is left in an odd state by the failed pairing on the C8.
I really don't really care who's "fault" it is. I am just hoping that the cause can be identified in case there is a way @gopher.ny can make an adjustment on the C8 to avoid this problem occurring.
Obviously there will be no updated firmware for these older Iris and Visonic devices.
I haven't kept good enough notes to be sure, but I don't think this issue appeared (or wasn't happening so frequently) until 2.3.5.135 when the multiple pairing choices were added (normal, avoid 3.0 routers, and exchange in clear), but I could be wrong about that.
My hub seems stable right now w/power at 16 on .135. What I should do to help figure this out is roll back to some previous releases and try joining the same devices to see if the problems w/reset occur to help narrow down when this started happening, but frankly my wife is so tired of the many "service interruptions" that I'm hesitant to mess w/things. I don't want her cancelling my contract due to failure to meet my SLA.

Come to the dark side
It's people like you who lead others to high power Zigbee settings, brownies w/ice cream and extra fudge sauce, and Amazon carts so full you have to scroll for five minutes to get to the bottom of the list.

That does seem to indicate to me that the device is left in an odd state by the failed pairing on the C8
I think something else must be going on; you got me playing with resets/joins again and I'm also seeing issues... I still contend that the only point of a device reset is to eliminate 'odd states'.... take this from an old hardware designer. If the device isn't doing that (truly resetting the hardware), it's broken (and FWIW, these things can break... the flash does wear out. There are lots of interesting posts on TI and SiLabs forums re: lack of wear leveling with Zigbee devices due to the number of times they write device parameters to flash that need to be retained during battery pulls)...
I just spent a few minutes observing the LED's (during reset/rejoins, and during simple battery swaps) on one of my Iris V2 contact sensors and came up with yet another head scratcher... resetting/rejoining an previously joined Iris V2 contact sensor would result in 'previous device found' as expected, yet the device wouldn't always register open/closed events on its details page a few moments later. Repeating the process (reset/rejoin) didn't fix the issue; it always was found immediately as previous device, usually worked for a few moments (though sometimes not??) then failed to register open/closed events. FWIW, it was joined near the C-8 (running .135) and appeared in its child table as expected.
Removing/replacing the battery (which normally is a non-event, requiring no interaction with the UI) resulted in a blinking blue LED; indicating it had failed to rejoin. Thinking that maybe the hub needed a reboot (though this hub has only 7 devices and no apps), I rebooted the C-8 but this didn't fix anything; resetting/rejoining this device (which had been in the device list for weeks) just wasn't working.
I then tried another Iris V2 contact sensor which wasn't in the C-8's device list; it joined normally as 'new device found' and continued to register open/closed events normally. Battery pull/insert resulted in normal operation (tried multiple times, all succeeded). Worked fine!
On a hunch, I 'removed' the problematic V2 contact sensor from the C-8, it joined normally as 'new device' and properly registered open/closed events.... and also rejoined normally with battery pull/insert (again, tried numerous times; all succeeded). All these resets/rejoins/new joins were done within a foot of the C-8; none went through a router 30' away.
So... what could the problem be? Perhaps some kind of database issue (something in the address table isn't getting mapped correctly during resets/rejoins) that is resulting in problems with previously joined devices. Seems kind of similar to the reset/rejoin issues that were found and fixed several releases ago (though in that case, reset/rejoined devices wouldn't give an indication in the UI that they rejoined... now they do, but have issues). Tagging @gopher.ny
@danabw can you try removing one of your problematic devices from the database and see if it functions properly when reset and joined as a new device? Of course your automations would need repairing....

I just spent a few minutes observing the LED's (during reset/rejoins, and during simple battery swaps) on one of my Iris V2 contact sensors and came up with yet another head scratcher... resetting/rejoining an previously joined Iris V2 contact sensor would result in 'previous device found' as expected, yet the device wouldn't always register open/closed events on its details page a few moments later. Repeating the process (reset/rejoin) didn't fix the issue; it always was found immediately as previous device, usually worked for a few moments (though sometimes not??) then failed to register open/closed events. FWIW, it was joined near the C-8 (running .135) and appeared in its child table as expected.
I have also seen that exact issue, more than once. In my case the device must be removed from the hub to be re-joined. All pairings as an existing device are as you note - they don't register events, the device really isn't re-joined. I've been using Swap Apps Device to replace the devices in my automations w/a virtual device of the same kind, and then removing the actual device, re-pairing it, and swapping it back into my automations.

On a hunch, I 'removed' the problematic V2 contact sensor from the C-8, it joined normally as 'new device' and properly registered open/closed events.... and also rejoined normally with battery pull/insert (again, tried numerous times; all succeeded). All these resets/rejoins/new joins were done within a foot of the C-8; none went through a router 30' away.
Yup - same, I have had to remove the devices from the C8 in order to enable an actual pairing of the device to the hub. The "previous device found" pairings do not actually work and events are not reported. Once I remove it and get it to pair the device behaves normally until the next time it falls off the hub.
My testing was consistent across multiple Iris v2's and Visonic sensors, so the likelihood of failed flash HW across multiple devices is very low.
Regarding the V2 oddball/can't reset properly state, is there any known way to force the C8 join to fail? such as the device enters this state?