C-8 Locks Up - Hard Crash - Every 4 to 9 days - Diagnostic Port DEAD too

🛎️ Get Help
1

Please provide your hub model (C7, C8, etc.) and its platform version from Settings>Hub Details.

Check out the following post for help troubleshooting problems and gathering details that will help others to identify and solve the problem you are experiencing: 🛎️READ FIRST - Before Posting in Get Help

I've looked at the posts from others with crashing Hubitat. Nothing helpful seen.
I a m on a C-8 with Platform Version 2.3.6.146
C-8 is wired to router.
C-8 has static IP from router.
C-8 is on a UPS as are all other network critical devices.
Started with a C-5 in 2020 or so. Migrated to C- 8 in March.
1 - Z-Wave device
10- - Zigbee devices
1 - Konnected Pro board, also on wired Ethernet.
I have a Synology Diskstation for network and for two IP cameras, one is usually off. Latest software.
I have a Synology router running latest software.

Never had a problem until now.

First crash was so odd, I just power cycled it from USB-A plug on power source.
Then it happened again on 2023-11-05
Again on 2023-11-09
And now again on 2023-11-18.
This is now serious. Once is happenstance, twice is coincidence, three times is enemy action.

On this last one, I tried getting into the diagnostics port. Nada. Nothing. Zero. Hubitat is DEAD. Green LED but DEAD.

Nothing stands out in the logs, just a time bump in the past logs from last event to the post power cycle events. Nothing stands out the logs.

Here are my Apps:
All Off hubitat
Battery level notifications elfege
Device Status Announcer RMoRobert
Device Status Announcer Child 3 RMoRobert
Homebridge v2 tonesto7
Hubitat Package Manager dcm.hpm
Konnected Pro konnected-io
Konnected Pro Device konnected-io
Temperature Delta Child mbarone
Temperature Delta Manager mbarone
Virtual Keypad Child mbarone
Virtual Keypad Manager mbarone
The most recently added one is Device Status Announcer, which runs at 22:00 - no where near the crash times.

My Drivers are:
Dahua/Amcrest Camera tomw MotionSensor, ImageCapture, Initialize
Hubitat Ping thebearmay Configuration, Actuator, Initialize, PresenceSensor
Konnected Beep/Blink konnected-io Actuator, Alarm, Switch, Momentary, Tone
Konnected Contact Sensor konnected-io ContactSensor, Sensor
Konnected Motion Sensor konnected-io MotionSensor, Sensor
Konnected Siren/Strobe konnected-io Actuator, Switch, Alarm
Pi-hole Virtual Switch CopyCat73 Switch, Refresh
Twilio Container mlritchie
Twilio Device mlritchie Notification, SpeechSynthesis
Virtual Basic Keypad mbarone LockCodes, Actuator, TamperAlert, ContactSensor, Momentary, PushableButton
Virtual Keypad mbarone LockCodes
Virtual Keypad Button Child mbarone Actuator, TamperAlert, Switch, Momentary, PushableButton
Virtual Keypad Input Display Child mbarone Notification

Twilio is useless crap since the 10DLC stuff and I've tried to fix it with them. Useless people.

I don't see any any correlation between the apparent crash log events and the device or apps running. I know the thing crashed when my sunset events fail to happen.

Until now the Hubitat has been absolutely solid, The C-5 and the C-8.
Now, I question it as too unreliable for home control and alarm control.

I'm thinking of trying out the Hubitat Rebooter app and having it restart every dew days.
But, that is a patch on a failure and I'd rather fix the failure.

I could try removing apps/drivers most recently added (by recollection).
Do any of my listed apps or drivers look dubious.?
They all work and many have been in use for years or months with no issues ever.

No evidence of resource problems. No network issues, It is the Hubitat and just the Hubitat.

What to do?

2

Do you have any device using Jumbo Frames on the same network segment as the C-8?

1 Like
3

The good news is that green means not dead. Do any of your automations using radio devices work? Like a Z-Wave/Zigbee light turning on when a Z-Wave/Zigbee motion sensor activates?

1 Like
4

Make sure jumbo frames is disabled, standard MTU. If it sends any multicast JF that hits the hub it can crash the network interface.

2 Likes
5

I sent the below by email, should have done it here...

On the MTU issue.
To be sure, I checked the router, NAS, and network switch.
This required some Internets searches to find where MTU is set.
But NO, jumbo frames are not set anywhere. Never has been set.
MTU is 1500 all around.
I read some postings on this before my posting.

From what my C-8 shows, green can in fact be DEAD.
No response on link to login screen, no response to diagnostic port.
No application actions performed.
Log file is empty for the period.
Dead. DED not worth spelling.

The dead period from last action shown at 16:02:33 was for a Konnected motion sensor (wired Ethernet).

Nothing from Konnected, Z-Wave temperature sensors, or contact sensors until I power cycled at 17:37:17
This is now the 4th such event, third since I started investigating.
All cases the same, Green and DEAD, but I did not check the diagnostic port until today.
I always first notice the problem when my sunset triggered lights are dark.

One suggestion in from Hubitat is to restore a saved configuration.
Seems like a waste of time, but I may try it. My configuration appears as expected and the little box runs swell until it doesn't and I've been checking it often lately.

I had absolute faith in Hubitat until this.
Three solid years with no incidents C-5 and then C-8. I've converted two friends to Hubitat.

Seeking alternatives or a fix. Looking for, but see no pattern other than it happens in the afternoon.

6

The symptoms you've initially presented, were indicative of a network issue, that's why I asked if automations are working, as they would, when the hub is not dead. The fact that the hub stops working, and you have no activity from the radios, that may be related to a hardware malfunction. If you'd like us to confirm, please create a warranty case by visiting the following page and we would be happy to further investigate if a replacement is required or if it is something that can be fixed with a software update:

3 Likes
7

I've had a very similar problem with my C8 a couple months ago. Hub would just lock up, green LED but no response on anything. Eventually I timed and traced the crash to happen every time it does a scheduled backup.

I fixed it by doing a Soft Reset and restore from backup via the Hub Diagnostic Tool. Still not sure what might be the problem but if all else fail, might be worth a shot for you.

1 Like
8

My post mentioned I was considering this, but thought little of it.
Never having ever done a Soft Reset and Restore before I was hesitant, but trusting Hubitat seemed to win the day.
I did the Soft Reset and Restore as you have done and suggested,

Time will tell. It was 9 days between the last two crashes, I'll have to wait weeks to to see if this helps. The Hubitat software did as it was supposed to do and everything appears as it did before.

Thanks for the reply.

9

To which email address? support@hubitat.com has been an invalid address for a very long time.

As @bobbyD recommended, if you are experiencing what may be a hub failure, the best way to reach Hubitat Support is to open a ticket via the support portal (support.hubitat.com).

3 Likes
10

I'm going to add a "me too" to this. Hub was pretty reliable. I've got it on wifi instead of a wired connection so jumbo frames shouldn't be an issue. Ever since I upgraded my c-8 to 2.3.6.146, it has crashed requiring a reboot. No response on the network and despite having a green light, I believe it was a hard crash because timed events didn't execute (we have it turn on a few of our zwave lights every evening). There was nothing of interest in the logs prior to the hangs... just normal log messages (often just reports of a temperature change from an ecowitt temp monitor which I have enabled) then nothing until the reboot. Should I go ahead and open a ticket too? I feel like this has worked fine until this latest update, so I'm not so sure it's necessarily a hardware issue (unless the timing is just coincidental).

11

Yes. And also perform a Soft Reset along with a database restore - as described in the online documentation.

1 Like
12

IIRC, Hubitat staff have confirmed that a local backup followed by a restore of the database backup just created automatically includs a soft reset...no longer necessary to do the soft reset as a separate step.

13

Hmmm...
I too never experienced Hubitat issues of any sort until sometime after the recent update or two.

Maybe our Hubitat overlords need to look into this. I'd be very happy to have a software update fix the problem and not deal with returns/exchanges because a hardware issue seems unlikely. In the years I've had my C-5 and now C-8, I have installed every offered update and never experienced an issue. I waited for 4 repetitions of this crash to post thinking it was my configuration.

I may have to keep track of updates along with my issues to look for cause and effect.

Good luck to all experiencing this.

14

Just did the reset. I also did some digging and I have seen past logs complaining about weewx getting spammy (I had it sending realtime pushes)... I've backed it off to no more than 30 seconds, just in case that was the cause of the problem (but it's also been like that for a while). I'll also open a ticket just so the devs can dig in and see if there's some sort of common issue.

2 Likes
15

We would be too, however, a hub freezing has only been observed when jumbo frames are enabled in the network. In most cases similar to yours, the problem was traced back to a local network issue. In very rare situations, a hardware malfunction was the problem, but those cases can be counted on one hand out of thousands of hubs of all models combined, not only C8, for example.

Historically, isolated incidents when an update such as 2.3.6. has been out for months, can't be corrected with a software fix, unfortunately. But never say never, right?

2 Likes
16

Roll back to a previous stable release until you feel comfortable updating? In the diagnostic menu, you can roll back to a previous release. There are limited numbers of update rollbacks (3?) but you might find a more stable version that way.

Not sure if you can get to 2.3.5, where you imply things were better (i.e. not crashing before 2.3.6), but might be worth seeing how far back you can go.

2 Likes
17

You can pick the last stable version of an older update by downloading to your hub from the following end point:

hub/advanced/downloadPlatform/xyz

*where xyz is the update version E.G. 232, 233, 234, 235 and so on.

5 Likes
18

My last action was to soft-reset the Hubitat and restore a backup configuration file.
I'm going to run that experiment for two weeks or until it fails.
Given that there is a suggestion that I do in fact have a rare hardware issue, I'll let that progress too.
I'll hold off on the option of testing a previous software release until one/both of these options play out. My experience is that Hubitat does a fine job of testing software for release and there would be many more complaints if it were the latest Hubitat release.

I have also checked to ensure my Raspberry Pi's cannot use jumbo frames without serious patching. All the other wired devices on my network go through a GC110P switch which has the MTU at 1500 so they cannot be a problem. Also, I'd think my Hubitat would suffer far more frequent issues if any device used jumbo frames. And none of them has a any reason to send packets to the Hubitat.

Thanks.

19

@kutoroff - Are you by chance tracking your free memory? While I expect the hub should be able to run longer than 9 days before getting tight on memory, I have, on occasion seen where a single event will cause a drop in memory availability and require a reboot earlier than normal.

1 Like
20

The only times I reboot my Hubitat, until very recently :-), was for software updates. It would run for weeks with no apparent or reported issues. Until this post and some replies, I knew of no method to view Hubitat resource use other to watch for alerts. I only see update available alerts.
A quick Google search turned up the "Hub Information Driver. Thanks for this.
I'll give it a try soon.
I think my Hubitat is lightly stressed, if at all.
But am curious and I want to know.

Thanks for the message and the handy driver.

SK!

1 Like