It is definitely something that is perplexing to us as well, but in trying to be fair, there are some signs that it isn't related to the device firmware. Firmware 2.15 for the device has been out since July of 2023. There are thousands of devices on all kinds of platforms and this issue (so far) is unique to Hubitat. The issue has only appeared suddenly and seems more related to a change on the hub than on the device. As @harjms mentioned there is a beta firmware that disables keep alive on Hubitat and the results have been promising. If we could release the beta firmware to a subset of users instead of making it available to everyone that would expedite getting it out to those that need it. I don't think that option is possible on Hubitat though.
The theory of @hydro311 about the fan switch is interesting. @dcaton1220 can you remove the fan switches from your network to see if that resolves the issue? I don't think it is related, but would be an easy way to rule that out.
That seems a reasonable conclusion. The C8 in particular has had its share of zigbee issues but in this case it may be coincidental. Hopefully the HE folks will respond.
Sure. I just air gapped them. If the problem remains, it ought to be evident later today, or in the morning at the very latest.
Well, this is the first 24 hours that my system has remained stable since sometime last week. I have six fan switches, all air-gapped so completely offline.
I don't believe I've updated FW on the fan switches; I don't recall any FW updates even being available. I did update the driver recently though. I have logging for Hubitat Package Manager turned off so I don't know the exact date, but I am running the driver dated 2024-02-16 so obviously it's been in the past 4 days. That seems to coincide more or less with the time I started having issues.
So what does all this mean? Something in the FW and/or driver of the fan switch is causing HE to burp and all the 2-1's end up going offline? That's wierd.
What else can I do to try and figure this out? No response from the Hubitat folks so far.
If the driver list says last modified 2/16 then that would be when it updated.
Turn off automatic updates on HPM and/or exclude Inovelli from the automatic updates. IMO that is a terrible feature to have (unless you only use it for specific packages you 100% trust).
@ericm should be able to get a direct link to the prior driver code from github. I would grab it for you but I am not familiar with their github and models.
Manually import the older code into the existing driver entry. That will roll back the driver.
I got the date from the comments in the driver itself.
I can get the old driver code from Github, that's not a problem. Want to see how Eric wants me to proceed first. Would probably be good if I could capture the specific event that's causing this issue, or at least starting the downward spiral.
Yes, the 2-1's (48 or so) and six fan switches. First few times it happened I did a factory reset on every one. I reset the Zigbee radio at least one or twice as well, which turned out to be unnecessary and didn't solve anything.
Eventually I realized that all the Inovelli switches needed was a power cycle. The last time it happened I simply turned off the main breaker for about 10 seconds, and they all rejoined the mesh on their own.
So, far as I know I didn't do anything to resolve the problem. It started seemingly on its own, and I have no idea if/when it will happen again.
This is stupid but actually looks like it may have worked. 24h latter my 2-1 are still connected. Well kinda. I flipped main breaker for about 5 minutes at 1pm Saturday. I woke up Sunday at 8am and my internet was working but also busted lol, was weird so I power cycled my network cabinet (router, hubitat and fiber box). That fixed the internet. 24h later still no issues
Yes there can be, air gap is only required to cut power to the load / fixture. So parts of the internal circuitry could still have power. I have seen cases before with Z-Wave devices where a full power cycle via the breaker has a different result than just air-gap.
Not sure if that's what you did differently which finally solved it (full power cycle via breaker)? It would be good to have that clearly posted for anyone else who stumbles in here looking for info.
This also leads me to believe that the full power cycle is what fixed it for this person as well.
A full power cycle is what resolved this for me, at least for the 2-1 switches. My fans have been on a beta firmware that fixed them but I based off the symptoms I believe the power cycle would have resolved for them as well.
So now, "Turn off all power to your house" will become a suggestion for dealing with Zigbee issues...
Seriously, when you air gap the switch the leds go out, which seems to indicate that all power, including the internal electronics, are disconnected. All may not be as it seems though, but then that begs the question why would some part of the electronics go out but some other part still receives power.
Just depends on how the device was built. Usually an air-gap will restart the device but it would not be inherently required for the air-gap to do so. Its possible part of the board still has power when you air-gap, if that air gap switch is connected to the circuit board vs directly on the incoming power feed.
