C-7 / C-8 and the potentially different batches of Inovelly Blue switches SoC (system-on-chip) may be the reason for the different experiences, even with one and the same firmware version.
2 Likes
Yep so few things have been established.
Happening on both a c-7 and c-8 hub
Happening to different batches (2-1 refurbished blues and blue fan controllers)
Happens every 12 hours, independent of other devices.
Airgaping the switch resets the 12h timer
Rebooting the hub/rebuilding the zigbee network does not reset the 12h timer.
Normally Iād agree, but myself and the user have no common denominators. He has c7, Iām on c8, different internet connections, itās every single blue on the network, some of mine are refurbished blues, some are brand new fan blue so different model switches. Definitely a weird issue. Something is happening every 12 hours. Be that the network trying to rejoin the devices or the devices trying to rejoin the network I canāt say but but somewhere somethings causing a disconnect 12 hours after the device connect
2 Likes
After 12 hours it seems that the switch is sending a keep-alive packet, but the process isn't quite working like it should for some users. Not sure if it is the hub or whether it is the switch. Haven't been able to find a common denominator yet but it is a small sample of users.
We have been testing a firmware that disables the keep-alive function when it is detected that the hub doesn't support it and that has looked promising.
6 Likes
Can confirm after sniffing packets the hub is not responding to the switches keep alive packet. The switches drop off the hub network and have to be airgapped to restore connectivity. Thanks to Inovelliās @ericm for troubleshooting this issue and getting a solution resolved in a timely manner. Issues go away after the latest switch firmware.
1 Like
@mike.maxwell I think you said up above you wanted to know if there was an issue on the hub side.
1 Like
Count me as a member of this unfortunate group. I saw this thread a few days ago, but didn't realize it described my problem as well. I've just wasted three days of my life rebuilding my zigbee mesh three times from scratch. Almost 90 devices.
I figured it must be something I'm doing wrong, or something went bad during the migration from C7 to C8-Pro (which seemed to work ok for a week or so).
I have 50+ Inovelli blues, which have been working correctly for the better part of a year now. All on the latest FW and drivers. I also have 6 blue fan switches that have been working fine for the past 4 or 5 months, also on the latest FW and drivers.
All of a sudden, nothing works more than a few hours. After getting everything back and running last night, I wake up this morning and the whole thing has turned to $#!+. Nothing works. How is this possible? The only things that have changed are the Inovelli drivers, and the hub FW.
If the switch is sending a keep alive packet every 12 hours, how was this not a problem until recently? Kinda points to a problem with the hub, which gets updates much more frequently.
I don't really care right now what part of the system is at fault, just how to get out of this nightmare. I'm really fortunate that my wife doesn't care when things don't work right, but after three days of this I'm ready to blow a gasket.
1 Like
To add to the fog & friction, it's not affecting all users -- my (9) Blue 2-1s have all been fine.
The one constant so far (I think) is that affected users have both Blue 2-1s and fan switches; I don't have any fan switches.
Glad I didnāt go this route as I wouldāve lost my mind after rebuilding the network to have it fail 12 hours later.
Iāve manually updated 6 switches with the alpha/beta firmware and the 6 switches have stayed connected for 4+ days.
I havenāt updated to the latest Hubitat firmware in case there was anything ANY of the manufacturers wanted to test, but ultimately Inovelli will push out the firmware to update the switches to support either hubs that support or donāt support the keep alive clusters.
Kinda appears that way. What I don't get is why it's affecting the blue 2-1's, which really haven't given me a drop of trouble in close to a year now. Perhaps the fan switches are doing something the hub doesn't like, and the hub (or the zigbee radio) throws a tantrum and borks the entire network?
I'm really kinda pissed off that all the blues seem to have gone into some non-functional mode because of this. I did a fair amount of research before I bought all this stuff and went with the blues and Hue retrofit downlights because they could be bound together and work without the hub being up and running.
That, so I thought, would be the closest in terms of reliability to hardwired lights: they would just work as long as AC power is available. I have been proven wrong. Thousands of dollars later, some wierd problem turns up and I can't even turn the lights on in my kitchen.
If I had money, I'd design zigbee or z-wave lights that worked with a conventional dimmer. They would turn on/off and dim without any wireless connection, but could also use zigbee or z-wave for color changes or more advanced functions. They would also be end devices, not repeaters, so removing power would not disrupt the mesh. A hybrid with the characteristics of conventional and smart bulbs.
My bindings still work even without the hub. Only if you need the hub to translate (e.g. scenes) should it cause an issue with the switches. Most (like 98%) of my 2-1s and fan switches are controlling dumb devices so the switches work without issue. I just can use rule machine or Alexa/homekit to control any of them (once they disconnect).
1 Like
As I mentioned, everything was working fine before I went to bed last night, and this morning it's all gone to hell. My mental state is definitely in question here.
It's not that my automations aren't working. I've always considered that to be icing on the cake, and not essential for daily life. It's when basic functionality doesn't work that drives me insane, and has me questioning the decisions I have made here.
It will get straigtened out one way or another, but I can't help thinking about what would happen if I met an untimely demise and this happened to my wife. She would not have a clue what to do, and I'll bet 99% of electricians wouldn't know what to do either, except to start replacing switches and lights. Even my brother, which is fairly tech savvy, wouldn't have a clue as he isn't into home automation.
Looks like this is my fault. When everything stopped working (or so I thought), I reset the zigbee radio and started over. Of course, this created new network ids and borked all my Hue lights at the same time, rendering the previous bindings useless. It didn't occur to me to power gap the Blues first and see if that fixed anything.
I just power gapped a couple of switches bound to some Hue downlights, and they do in fact work. So clearly the issue here is at least partly due to the Blues not reconnecting to the mesh, and why they disconnected in the first place.
But it would appear that the temporary fix for the problem is simply to shut the main to the house off for a few seconds.
1 Like
Ahh yes. I did that too and was like wtfffff. But then I remembered I factory reset some switches and needed to rebind them.
Just know the fixes are on the way. Caveat is youāll need to update them within the 12 hours after air gapping them (before they disconnect from the hub again). Otherwise youāll need to air gap again and start again.
Well, the brute force method of shutting down the main breaker for a few seconds brought back everything from the dead (safe, since all critical stuff is on UPS). Wish I knew that three days ago... 
From who? Has the root cause been identified? Kinda points to a HE issue. The blues haven't had new firmware in months, so why now is the keep alive thing such a big problem?
Doesn't seem to be isolated to the C8 either, which seems to rule out a radio chip or radio FW issue.
@mike.maxwell Any update on this?
Zigbee is more @mike.maxwell's territory than Victor's.
It's certainly possible this all somehow involves HE too, but the smokiest gun so far is an existing 2-1 setup where Fan switches were added, so that points to something with the Fan switches.
I love my Blues and I'm a big Inovelli fan, but to me, this sounds like something in Inovelli's code -- this wouldn't be the first time an intentional-but-non-standard (for lack of a better term -- I don't mean to be inflammatory here) zigbee implementation of theirs caused unexpected results...
@dcaton1220 -- you remember this saga from late last year. The 2023-12-22 line fix in the 2-1 driver finally patched that issue for us.
Just to clarify from my side my zigbee network was built out to include fan and blues. I never an existing network then added fans to it. No blues of either type were added in the short term before issues started
Fair enough -- I should've just phrased it as "a network with both 2-1s and fan switches" seems to be the common factor thus far.
May very well be, but I don't get the connection between the fan switches and the 2-1's. Even if the fan switch is doing something wrong, it shouldn't prevent the other switches from working.
That issue wasn't really my problem, although I thought it was. The problem I had (and still do) is how to bind a switch to a group so that it responds to that group's messages, but doesn't initiate any on its own.
In an attempt to reduce zigbee traffic and dropped messages, I wanted to create a group with a large number of switches that I would use to turn them all off with a single group command. Fortunately, I only tried this with a few switches because it had unexpected consequences. The switches did all respond to an off command sent from the hub but turning any one switch on or off turned all of the others on/off as well.
I guess this is how it has to work, otherwise you couldn't have a switch control a group of Hue lights, for example. Maybe binding the switch to a group is the answer, rather than adding the switch to the group, but there's no way to do that in HE without writing a special app or driver. I gave up on that for now.
If it was solely an Inovelli issue, I feel a lot more users would also experience issues too; not just Hubitat hubs. My hub in the garage 25 ft away is still running solid during this initial post; granted I only have a few 2-1s connected to it.
Iāve been running the alpha (mg21), beta, and released hardware for the fan switches for over a year. I donāt see the connection outside of recent firmware updates on both manufacturers. It just randomly stopped working.
Edit: the mg21 firmware hasnāt been updated since the mg24 fan switch and firmware was developed so itās not even like I can say itās a Inovelli firmware issue.
1 Like
