[INCIDENT] Cloud backup failure leading to unresponsive Z-Wave devices

Over the past few months, our engineers have been closely monitoring incidents of cloud backup failures that caused Z-Wave devices to become unresponsive.

We’re excited to announce that they’ve not only identified the root cause but have completely re-engineered the Z-Wave backup process.

While occasional backup failures may still happen, the great news is that these should no longer lead to unresponsive devices or require users to power cycle the hub to restore Z-Wave functionality.

This revamped process is now live in the beta channel, and if all goes smoothly, it will be available to everyone soon! Stay tuned for the full release or check out this post to learn how to join our Beta Program, today: Join Hubitat Beta Group

28 Likes

Yes, great news. I had to teach the family to power cycle the hub when it was happening. this is great news

3 Likes

Anything interesting?

3 posts were split to a new topic: Broken Z-Wave

A post was split to a new topic: Z-wave thermostat stopped reporting updates to hub

C-8 running 2.3.9.197
Woke up this morning to non-responsive Z-Wave.
Message: Last cloud backup failed. Failed steps: Z-Wave.
Graceful shutdown, unplug for a minute, and restarted - seems to be working now.
I know others have seen this and I thought there was a fix - this is the first time I have had it happen. I had this hub set for weekly cloud backups because of the issues reported by others.

You just "got lucky". :slight_smile:

C-8 Updated to 2.3.9.200 and ran a manual cloud backup.

"Last cloud backup failed. Failed steps: Z-Wave.:

"Last cloud backup was unsuccessful. Failed steps: Z-Wave. Please power cycle your hub by shutting it down from Settings, then unplugging the power for 30 seconds."

I tried a Z-Wave device and it worked - I have not powered down again. I will monitor today.

I know these changes were backed out.. have they been re-integrated?

They were not backed out. The backup may fail occasionally, but the expected outcome is that it doesn't take down the radio requiring power cycling the hub.

4 Likes

C-8 after updating to 2.3.9.200 this morning I tried a manual backup to cloud. It failed but Z-Wave appeared to continue functioning.
I just tried another manual backup to cloud and that succeeded.

The alert message is still indicating the failure. Should that clear when a cloud backup is successful?

1 Like

Yes, it should. You can try running another backup.

Sometimes takes a few minutes to clear.

3 Likes

That must be it. It cleared while I was messaging Bobby.
Thanks!

1 Like

Hub is (rightfully) scared of Bobby. :wink:

7 Likes

:rofl:

1 Like

Thankfully, my cloud backup ran last night without issue on .200

1 Like

I also had a cloud backup failure today (regular weekly backup). C-8 Updated to 2.3.9.200 a few days ago. I don't know if the z-wave devices were "unstable" after this as I saw it first thing this morning. I have an automation (in Node-RED) to shut down and restart the hub (not reboot) that I kicked off. Did a manual backup after that was successful. So far nothing out of the ordinary :crossed_fingers:t3:

EDIT: The notification of the failure still says to shutdown and restart the hub. Is that intentional?

I believe the wording change is waiting for verification that it is functioning as intended in the real world and not just the beta group.

2 Likes