Beyond Frustrated with C8 Migration

[dustin.carpio]

I have been a Hubitat evangelist for years. My C5 was the backbone of a very well-oiled smart home and had my wife's and kid's acceptance factor. However, after the C8 migration, that has all changed.

My daily schedule now includes resetting many Zigbee devices, this is especially true if the Zigbee device has a battery in it. My wife now laughs whenever I need to get up on a stool to reach a motion sensor that isn't working. The craziest thing is that there seems to be no rhyme or reason to why a Zigbee device falls off the network. I could walk into a room, have the motion sensor pick me up, and then walk out 5 minutes later and the Zigbee motion sensor won't report that motion stopped until I remove the battery and power it back up. This just did not happen EVER with my C5. Something is very wrong with the Zigbee radio implementation on the C8. It took me 3 hours to get my Ikea blinds to re pair with the hub yesterday. They have literally never gone offline on my C5.

The surprise this week was that my once very solid Z-Wave network now can't communicate with half the devices unless I power them down and back up. Even then it's a crap shoot on if the device will rejoin the network. Sometimes the communication is so delayed on the ZWave network that a button push will take 90 seconds to register and turn on or off a device. Craziness.

This morning, I woke up to a nearly unresponsive hub. Apparently, the memory had run out for seemingly no reason. This never happened on the C5 and I have not added any additional automations or devices since the migration to the C8.

I love Hubitat, but the C8 has made my smart home unusable. I can't be the only one who is willing to pay a monthly fee to get these problems fixed. Please, this madness needs to stop.

[mavrrick58]

It is totally understandable. It seems like getting any new hub can be a bit rough after it's first launch. I read so many threads when I got my C7 about how unreliable it was and how it so many Zwave issues. I personally never experienced them but saw allot about it. This time Zigbee devices are being hit as the C8 has a whole new stack for Zigbee as well

My first suggestion would be to ensure you are on the latest firmware. There have been constant tweaks to the Zigbee stack on the C8 and unfortunately one of the most recent Firmware released actually force users to the new stack. It seems some users were using the old zigbee communication stack somehow even though they shouldn't have been. Being on the latest firmware should clear that up.

Then I would suggest a full power down for 1 min to clear the radios and then restart. I have found a full power cycle with the Zwave radio being reset can do wonders when zwave starts to miss behave.

Lastly how did you know your memory was low. Do you track memory with the Hub Info driver or through Node-Red. I have seen one occasion where my hub dropped extremely low with memory, but it is suppose to recover in about 10 min.

Lastly a question about your gear. Do you use any devices that are prone to have issues with Hubiat like Aquara or Xiaomi devices that need special repeaters to function.

[dustin.carpio]

Thanks for the suggestions. I'm very prompt with updating to the latest firmware. In fact, I'm updating to 124 right now.

I will try to prolonged power down when I get home today. I have seen that clear up issues before so maybe I just need to be more patient.

I know my memory was low because when I logged in to see why nothing was responding this morning my C8 hub told me the memory was low in the notifications area.

I don't run any non-standard gear. I have 33 ZWave devices and around 45 Zigbee devices. All have either officially supported drivers or community drivers from HPM installed. The only oddball out is the Harmony driver, but that never has issues. All of these drivers worked flawlessly on the C5 hub so I think the updated radio stacks are the real culprit here.

[terminal3]

I am a little frustrated with it myself. My system is pretty stable, but my plan was to switch to a single device hub in the house so I could move my C7 hub out to the shop. Since hearing about all the Zigbee issues and the fact that I have several Iris V1 devices that will not pair to the C8 I have stopped the process. Until it seems the Zigbee issues are worked out I will just have to maintain the two device hubs, one for Zigbee and one for Z-wave.

[mavrrick58]

@dustin.carpio

I know my memory was low because when I logged in to see why nothing was responding this morning my C8 hub told me the memory was low in the notifications area.

If this is how you found out your hub was certainly struggling by the time this was found. did you wait any amount of time to see if it would recover on it's own, or did you just restart the hub? If the memory gets low enough you will most definitely see issues with Zigbee and Zwave.

You can see in this image of my prod hubs(C8) Memory usage it dropped down to as low as 109MB at 4:42 this morning. It recovers immediately though. I think something they did in the last release does allow the hub under some circumstances to really crush the memory for a short time for background tasks. How long had your hub be up when it gave you that message of low memory.

@dustin.carpio

All have either officially supported drivers or community drivers from HPM installed.

Unfortunately, this doesn't ensure that the device is one that will work well with Hubitat. This is why i called out Aquara and Xiomi as they tend to follow non standard Zigbee protocols on some of their devices and are known to be unreliable under certain circumstances.

[dustin.carpio]

The low memory alert remained for 30 minutes until I finally decided to restart the hub. I also got a notification that the nightly cloud backup failed to upload but I know my internet service was fine and there are no firewall rules blocking hubitat services.

I'm debating on whether or not to completely tear apart my system and rebuild it, but that would be very painful and result in many bad words. I run webcore and some rule machine legacy automations that I have yet to upgrade to modern rule machine.

[mavrrick58]

To me it makes sense that if your memory is very low the backup would likely fail. It is very clear based on my stats that the Backups weather it is local our to a external system causes significant memory drop when it occurs. That graph I showed above is when my Unraid server connects to my hub to create and download a new backup.

How long had it been between the hub being rebooted and getting that low memory error.

You may want to look at what may be creating significant memory load on your hub. I don't know if you need to go as far as tearing it apart and setting up again, but certainly a memory issue seems to be involved.

[neonturbo]

@dustin.carpio

My daily schedule now includes resetting many Zigbee devices, this is especially true if the Zigbee device has a battery in it.

Mine has been better with the 124 update. My setup was like yours, I was losing battery devices daily for the past few releases. I have been on 124 for a couple days now and I haven't seen one dropoff (yet?).

[danabw]

@dustin.carpio

whether or not to completely tear apart my system and rebuild it

I know the current situation is painful, but I'd hold a bit on that. Recent FW releases have generally improved things for many (most?) users, so we do seem to be making overall progress and possibly have some light at the end of the tunnel. (At least at the end of the largest part of the tunnel, to stretch a metaphor.)

TL;DR: I think we're getting close (or at least closer) to stability for most users.

[dustin.carpio]

@danabw - Your comment certainly makes me feel a lot better!

I will say that, after upgrading to 124, I only had to reset a few centralite zigbee motion sensors today. For the first time in 2 weeks, all my zigbee repeaters stayed online which made my Hampton Bay Zigbee fan controllers work perfectly.

I still have some zwave latency issues but that seems to have improved after a zwave repair. I noticed a bunch of zwave devices were finally making logical jumps through repeaters and always on zwave switches instead of bouncing around my house to tons of devices.

Whatever you did to the zigbee and zwave stacks seems to have made battery-powered endpoints reachable again. I will monitor for better performance over the next week and report back.

[danabw]

@dustin.carpio

Whatever you did to the zigbee and zwave stacks

I think the only affect I've had on them was from burning incense and chanting. I'm not a coder by any means, it's been HE staff mike.maxwell, gopher.ny, bcopeland and others on the HE staff who have been working mightily to turn water into wine, so to speak.

@dustin.carpio

I still have some zwave latency issues but that seems to have improved after a zwave repair. I noticed a bunch of zwave devices were finally making logical jumps through repeaters and always on zwave switches instead of bouncing around my house to tons of devices.

The Z-Wave mesh/devices should to improve their connections over time. If you have a specific device that appears to have janky routing you can try to nudge it via a Repair just on that device, but in some cases the device wants what it wants and will persist in routing we just don't get...in which case the old if it ain't broke rule will apply.

Oh, and if you're feeling lucky. 2.3.5.125 is out.

[Din.Djarin]

@danabw

2.3.5.125

I updated to 2.3.5.125 and I'm still having issues with adding 3 Iris v1 Zigbee motion sensors back the network. Otherwise everything else has been great for me.

[danabw]

@Din.Djarin

having issues with adding 3 Iris v1 Zigbee motion sensors

I don't think they've been able to address the Iris v1 joining yet while they've been dealing w/the messier Zigbee issues. @mike.maxwell will need to confirm where that sits.

@Din.Djarin

Otherwise everything else has been great for me.

That's great to hear...seems like most of the other serious issues have been resolved for most users. Most of most is pretty good, right?

[mike.maxwell]

@danabw

address the Iris v1 joining yet

I've looked at this a few times so far but without any success.
Once we get done messing with and getting currently available devices working acceptably i will look into this again.

[danabw]

@mike.maxwell

getting currently available devices working acceptably i will look into this again.

Makes sense. Thanks for the update, Mike.

[danabw]

@Din.Djarin

I'm still having issues with adding 3 Iris v1 Zigbee motion sensors back the network.

One option you have, if missing the motion sensors is a significant issue for you, is to bring the C7 online, and as soon as you do so reset the Zigbee and Z-Wave radios to clear all the devices from them. Then change the Zigbee channel on the C7 so that it's on 15, 20, or 25 (whichever is not the same as the C8).

Then you can add the v1 devices to your C7 and share then to the C8 via hub mesh. That's what I'm doing w/a few v1 devices - they still add fine to the C7. Even once the v1 issue gets fixes on the C8, you may end up liking having a second hub running...

[dustin.carpio]

Well the good news only lasted for a few days. I came home to find my zigbee blinds and motion sensors unresponsive again. So I'm now re-pairing EVERY SINGLE battery-powered zigbee device. It looks like everything went offline around the same time in the middle of the day, so it's certainly the hub not being able to reach them for some reason.

Not sure what needs to be done to roll back to a stable zigbee solution here but I'm getting really tired of the hubitat induced anxiety attacks. Please advise.

[danabw]

Take a look at your hub logs, they're on the logs tab under Hub and see if you see Zigbee offline event or events around the time when all the devices fell off.

[terminal3]

@mike.maxwell

I've looked at this a few times so far but without any success.

I am not that upset, but a bit that there wasn't some kind of warning the V1 devices might be an issue. I broke my own rule of never buying just released hardware. I should have waited. I am not beyond frustrated but slightly.

I understand you might at some point have to drop support for the V1 devices and if you had said we can't really support the V1 stuff on a C8 that would have been perfectly acceptable. Maybe there was something posted that indicated the V1 Zigbee might be a problem but I never saw it. Some stuff is just not forward compatible, I get that. If I had know, I would not have purchased a C8 when I did. I probably would have at some point. Either after you got the V1 devices working, or I had replaced all the V1 stuff with updated devices, I still have too many V1 devices in my system to drop them now.

Hopefully things will be worked out soon.

[dustin.carpio]

Well this would explain a lot! Look how many times the zigbee radio went on and off today alone. It's probably playing havoc with my older zigbee devices that don't recover very well.