zwaveCrashed 3rd time since December 2022

ronv42 · March 8, 2023, 10:47am

Woke up this morning to no z-wave devices working. Went into my C7 and had the dreaded zwaveCrashed Z-Wave unresponsive message. It was working just fine up to 4:30 this morning. In the middle of shutting down the hub right now and removing power from it.

My Z-wave devices and network hasn't changed in 3 years. I would say I have had this happen at least two other times since the end of December. I do remember that the hubitat guys were going to add more "logging" for these types of events but haven't seen any additional Z-Wave diagnostics other than the Z-Wave network display.

Any ideas of how to dig deeper into this? An no I don't have any "Ghost" devices, Here is my hub version. Since I can't seem to reproduce I re-ran all automations between 10:30 pm and 4:30 am after the reboot and the radio is working.

Here is the Z-Wave Device Table:

Unknown Title

Node	Stats	Status	Device Class
0x07 (007)	PER: 0, RTT Avg: ms, LWR RSSI: 55dB
Neighbors: 5, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Office-Lights	None	01 -> 0A -> 08 -> 07 100kbps
0x08 (008)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 7, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Outdoor-Front Lights	None	01 -> 0B -> 07 -> 08 40kbps
0x09 (009)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 5, Route Changes: 0	OK	SPECIFIC_TYPE_NOTIFICATION_SENSOR
Elexa Consumer Products Inc.	Laundry Room Leak Sensor	None	01 -> 09 100kbps
0x0A (010)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 7, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Living Room-Mantel	None	01 -> 0A 100kbps
0x0B (011)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 8, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Hallway-Lights	None	01 -> 0B 100kbps
0x0C (012)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 5, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Dining Room-Table	None	01 -> 0D -> 27 -> 0C 100kbps
0x0D (013)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 11, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Kitchen-Counter	None	01 -> 0D 100kbps
0x0E (014)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 9, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Dinette-Table	None	01 -> 0A -> 0E 9.6kbps
0x0F (015)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 5, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Family Room-Mantel	None	01 -> 0D -> 0F 100kbps
0x11 (017)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 7, Route Changes: 0	OK	SPECIFIC_TYPE_NOTIFICATION_SENSOR
Elexa Consumer Products Inc.	Basement-Leak Sensor	None	01 -> 11 100kbps
0x13 (019)	PER: 0, RTT Avg: ms, LWR RSSI: 13dB
Neighbors: 7, Route Changes: 0	OK	SPECIFIC_TYPE_NOTIFICATION_SENSOR
Elexa Consumer Products Inc.	Garage-Door Sensor	None	01 -> 1F -> 21 -> 13 100kbps
0x14 (020)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 19, Route Changes: 0	OK	SPECIFIC_TYPE_SIREN
Elexa Consumer Products Inc.	Kitchen-Siren	None	01 -> 0B -> 14 40kbps
0x15 (021)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 8, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Foyer-Chandelier	None	01 -> 0D -> 15 100kbps
0x18 (024)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 4, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
NIE Technology Co., Ltd	Master Bedroom-Lights	None	01 -> 0A -> 1F -> 18 9.6kbps
0x19 (025)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 18, Route Changes: 0	OK	SPECIFIC_TYPE_SECURE_KEYPAD_DOOR_LOCK
Allegion	Front Door-Lock	S0	01 -> 0A -> 19 40kbps
0x1F (031)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 15, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_BINARY
Honeywell	Garage-Light	S2 Authenticated	01 -> 1F 100kbps
0x20 (032)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 11, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
Honeywell	Outdoor-Back Light-Kitchen Patio	S2 Authenticated	01 -> 20 100kbps
0x21 (033)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 6, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
Honeywell	Outdoor-Back Light-Family Room Patio	S2 Authenticated	01 -> 1F -> 21 100kbps
0x26 (038)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 6, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_MULTILEVEL
Willis Electric Co., Ltd.	Stephanie Bedroom-Lamp Outlet	None	01 -> 1F -> 26 100kbps
0x27 (039)	PER: 0, RTT Avg: ms, LWR RSSI:
Neighbors: 18, Route Changes: 0	OK	SPECIFIC_TYPE_POWER_SWITCH_BINARY
Honeywell	Basement-Stairs	None	01 -> 27 100kbps

bcopeland · March 8, 2023, 10:56am

Can you give the output of /hub/zwaveVersion ?

ronv42 · March 8, 2023, 10:57am

VersionReport(zWaveLibraryType:7, zWaveProtocolVersion:7, zWaveProtocolSubVersion:18, firmware0Version:7, firmware0SubVersion:18, hardwareVersion:1, firmwareTargets:1, targetVersions:[[target:1, version:7, subVersion:15]])

bcopeland · March 8, 2023, 11:00am

Ok.. since you have already updated to 7.18 firmware.. Try this:

/hub/advance/installZipgateway718

Then when it's done.. Reboot ..

I had this happen to 1 other person and this fixed it for them

ronv42 · March 8, 2023, 11:06am

It looks like it does something quick then a 404 message comes up is that normal?

bcopeland · March 8, 2023, 11:09am

Ahh.. no 404 means it's the wrong url... and I see I had a typo there.. I apologize, I haven't had coffee yet..

/hub/advanced/installZipgateway718

ronv42 · March 8, 2023, 11:11am

Still 404, coffee needed

bcopeland · March 8, 2023, 11:14am

Ok.. Odd .. This is the correct endpoint.. are you sure you got the uppercase Z in there, it is case sensitive .. If it still doesn't work can you screenshot the url you are using in your browser?

ronv42 · March 8, 2023, 11:15am

bcopeland · March 8, 2023, 11:16am

Ok.. I see the problem .. you missed the /hub/ part

bcopeland · March 8, 2023, 11:17am

http://<your-hub-ip>/hub/advanced/installZipgateway718

ronv42 · March 8, 2023, 11:17am

Rebooting. Yep the green /hub I assumed was the hub's address.

bcopeland · March 8, 2023, 11:17am

Not sure why discourse was coloring it green .. but I can see why that was causing confusion..

ronv42 · March 8, 2023, 11:19am

Well time will tell now. Anything I should be looking for if it happens again?

bcopeland · March 8, 2023, 11:21am

Not really.. Just let me know if it crashes again

ronv42 · March 8, 2023, 11:23am

Thanks, I will let you know.

jtp10181 · March 8, 2023, 1:45pm

Ooo this updates what is listed as the SDK version in Hub Info. Just did it on my old C7 for S&G. Now that horrible 7.15 is gone. No idea what it does but now I'm updated!

bcopeland · March 8, 2023, 3:18pm

I should note for anyone else that picks this up.. Make sure your Z-Wave firmware is up to date first.. There is a bug in 7.18 zipgateway that was preventing firmware updates on 700 series, which is why I rolled it back..

jtp10181 · March 8, 2023, 3:24pm

Ah, I was wondering why I had 7.15 and I saw other people have 7.18, got the upgrade jealousy. Must have newest version!

ronv42 · March 28, 2023, 10:24am

@bcopeland This morning again the Z-Wave radio was not operating on my primary C7 hub. That is after we put in the the new Z-Wave gateway. I don't know what time it crashed but I assume it was around the nightly backup since the backup also failed and the backup's to the cloud include Z-Wave devices.

I am shutting down and doing the power cycle at 5:20 AM central right now that should bring it back. This is now the 4th time since December I am wondering if I should do a swap between my development Hub and this Hub with a full backup and restore to restore the Z-Wave on the Dev hub. But that would also mean having to re-include the ZigBee devices on the Dev Hub. This may point to a Z-Wave radio that is starting to go bad in the current production hub. Thoughts?

Here is the CPU and Memory chart of over night and you can see right around 2:30 a spike in CPU and a drop in free memory. I don't know if that is the DB cleanup or somehting else but backup's are scheduled for 2:15.