New C8 keeps crashing/locking up

I'm not sure what's going on by my new C8 keeps crashing. The green light is on, but cannot get access to it. No apps run from what I can gather, and there's no network access either (eithernet or WIFI). Can't see the C8 when I check my router... and yes I tied resetting the C8 network by pressing the reset button under the hub for 8 secs. I'm forced to again power cycle to regain funtionality.

It crashed/locked up twiced yesterday alone. I powercycled it just this morning and quickly loaded up the logs. I don't see any error when it crashed. Any thoughts? My C7 never experienced anything like this.

Here is a screen shot before/after powercycle.

What is device 226?

It's a zigbee outlet... Smartthings Smart Outlet

Any chance you have or had jumbo frames enabled on your LAN? Could any device be configured for JF and sending out multi-cast JF packets? This will crash the hub if JF hits the network interface.

I've never enabled it, nor am I familiar with it. The only thing I did this morning was to disable QoS on my ASUS router... which was active without issues on my C7.

I am using QoS on ASUS as well, does not cause any issues.

The only other thing that would usually take out a hub like that would be some sort of rouge app or driver. Usually you would see errors in logs or it would be apparent on the Apps/Device stats pages. Do you have any Wifi devices with user integrations? Those are most likely to cause problems.

Other than that nearly every incident like this the OP finally figures out that it was in fact jumbo frames causing it. Just had one yesterday come back after 2 weeks and said he finally found the machine that was configured for jumbo frames, turned it off, and crashing stopped.

Are you on the latest build?
I had a similar issue that would cause my hub to reboot or occasionally just lock up. Had to power cycle it, like you, to get it back.
There was a fix in the latest release to cure the issue.
Just a thought.

Yes as far as I know... 2.3.5.113

Yes. That's the latest. Guess it's not the same issue I had then. :thinking:

I'm looking at prior events in the logs up to the crash and see no errors except one which was at 11:06pm due to a disconnected device (automatic sprinkler off for the winter).

Watch the Device/App stats pages. Enable all the columns. Looks for things with high usage. The % of total is the most telling, most well behaved stuff will stay under 2% there.

So far devices are extremly low. Checked APPS and sorted from high to low...

If that MyNextManager is not throwing a bunch of connection errors I assume it should be OK. If there is any IP's configured in it make sure they are all correct and not trying to connect to dead or incorrect IPs. If you want to test it out you can disable the app temporarily to see if it makes a difference. How to disable apps: Apps Page | Hubitat Documentation

For giggles check on your router under LAN>>SWITCH CONTROL you will see ENABLE JUMBO FRAME. Check that it's off. Also maybe disable that device to see if things improve.

1 Like

Confirmed... it's been off.

Screenshot 2023-03-19 143608

1 Like

Good just wanted to eliminate that as a possibility

Now temporarily disable that particular device.

Can you also post your z-wave details page? (The whole thing)

1 Like

FWIW I'm currently migrating WebCoRE from USER to the built-in app. Just finished. I'm suspicious if WebCoRE may be the problem since going from C7 to C8 even though I've done a repair in HPM.

1 Like

I'm having this issue as well. It always seems to happen overnight but I can't verify that the time has anything to do with it. It's probably happening once every couple of weeks.

In the below logs, you can see that it disassociates at 3:01:36 and then attempts to re-associate, but apparently the handshake never completes and the AP drops (deauths) the client.

You can then see at 9:57 when I power cycled the hub.

There's nothing in the hub logs around that time.

In addition to the device being disconnected from the wifi, it seemed to be otherwise unresponsive - I have a Z-wave scene controller that I attempted to use and it did not work.

Obviously I wasn't able to get to the web interface or the thing on 8081 since it was off the network.

I do have jumbo frames enabled on my network, but that will require a lot of reconfiguration of other devices if I disable it, so that's not going to happen. I do not have problems with any other devices due to jumbo frames and I don't have any other devices with wifi connectivity issues.

The hub is probably 10 feet from the access point it's using so range isn't an issue.

I am currently running 2.3.5.135, I do see there is a fix related to wifi in 2.3.5.138 so I'll update it. I don't log into the hub very often, so every time I have encountered this issue, it turns out I'm a version or two behind :wink:

I've redacted the mac addresses. I pasted a screen shot of the logs below as well as a copy of that data in CSV format.

timestamp,source,message
2023-05-15T03:01:36.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ IEEE 802.11: disassociated"
2023-05-15T03:01:36.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ IEEE 802.11: sta_stats"
2023-05-15T03:01:36.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: : wevent[2012]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath0: @@MAC@@ / 4"
2023-05-15T03:01:38.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""sta_leave"",""assoc_status"":""0"",""event_id"":""6"",""avg_rssi"":""-65""}"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/4 msg of 4-Way Handshake"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: : wevent[2012]: wevent.ubnt_custom_event(): EVENT_STA_JOIN ath0: @@MAC@@ / 11"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1813850.279889] ath0: [@@MAC@@] station associated at aid 11: short preamble, short slot time, QoS, HT20 cap 0x1431"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1813850.279828] ieee80211_update_node_ecm_flags: node with aid 11 and mac @@MAC@@ has been tagged [ dvlan ]"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ IEEE 802.11: associated"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA f0:9f:c2:2d:c9:ec DRIVER: Send AUTH addr=@@MAC@@ status_code=0"
2023-05-15T03:01:48.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1813850.201838] ath0: [@@MAC@@] recv auth frame with algorithm 0 seq 1"
2023-05-15T03:01:49.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/4 msg of 4-Way Handshake"
2023-05-15T03:01:50.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/4 msg of 4-Way Handshake"
2023-05-15T03:01:51.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/4 msg of 4-Way Handshake"
2023-05-15T03:01:52.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: : wevent[2012]: wevent.ubnt_custom_event(): EVENT_STA_LEAVE ath0: @@MAC@@ / 11"
2023-05-15T03:01:57.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ IEEE 802.11: deauthenticated due to local deauth request"
2023-05-15T03:02:03.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""failure"",""assoc_status"":""0"",""wpa_auth_failures"":""1"",""assoc_delta"":""80000"",""auth_delta"":""0"",""event_id"":""7"",""auth_ts"":""1813850.216412""}"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/4 msg of 4-Way Handshake"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: : wevent[2012]: wevent.ubnt_custom_event(): EVENT_STA_JOIN ath0: @@MAC@@ / 11"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1838794.449362] ath0: [@@MAC@@] station associated at aid 11: short preamble, short slot time, QoS, HT20 cap 0x1431"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1838794.449298] ieee80211_update_node_ecm_flags: node with aid 11 and mac @@MAC@@ has been tagged [ dvlan ]"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ IEEE 802.11: associated"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1838794.377570] ath0: [@@MAC@@] recv auth frame with algorithm 0 seq 1"
2023-05-15T09:57:32.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA f0:9f:c2:2d:c9:ec DRIVER: Send AUTH addr=@@MAC@@ status_code=0"
2023-05-15T09:57:33.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: pairwise key handshake completed (RSN)"
2023-05-15T09:57:33.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: received EAPOL-Key frame (4/4 Pairwise)"
2023-05-15T09:57:33.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 3/4 msg of 4-Way Handshake"
2023-05-15T09:57:33.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: received EAPOL-Key frame (2/4 Pairwise)"
2023-05-15T09:57:34.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: : wevent[2012]: wevent.ubnt_custom_event(): EVENT_STA_IP ath0: @@MAC@@ / 10.1.3.217"
2023-05-15T09:57:34.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1838796.393797] [DHCP-SM] @@MAC@@: r-40-a"
2023-05-15T09:57:43.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ RADIUS: starting accounting session 6CF1FB61FB24BFBC"
2023-05-15T09:57:47.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""soft failure"",""assoc_status"":""0"",""ip_delta"":""2020000"",""ip_assign_type"":""dhcp"",""wpa_auth_delta"":""970000"",""assoc_delta"":""70000"",""auth_delta"":""0"",""event_id"":""1"",""auth_ts"":""1838794.392898"",""avg_rssi"":""-62""}"
2023-05-15T09:58:14.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""traffic_delta"":""41770000"",""dns_responses"":""2"",""ip_delta"":""2020000"",""ip_assign_type"":""N/A"",""wpa_auth_delta"":""970000"",""assoc_delta"":""70000"",""auth_delta"":""0"",""event_id"":""1"",""dns_resp_seen"":""yes"",""avg_rssi"":""-63""}"
2023-05-15T09:58:34.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-64""}"
2023-05-15T09:59:14.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-66""}"
2023-05-15T09:59:37.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-68""}"
2023-05-15T10:00:07.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-65""}"
2023-05-15T10:01:37.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/2 msg of Group Key Handshake"
2023-05-15T10:01:38.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: group key handshake completed (RSN)"
2023-05-15T10:01:38.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: received EAPOL-Key frame (2/2 Group)"
2023-05-15T10:05:04.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1839246.454386] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a"
2023-05-15T10:38:25.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-64""}"
2023-05-15T10:39:39.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1841320.718799] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a-406000-r-30-a-419030-r-20-a-411010-r-90-a-421040-r-30-a-417010-r-20-a"
2023-05-15T10:46:31.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1841732.757659] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a-406000-r-30-a-419030-r-20-a-411010-r-90-a-421040-r-30-a-417010-r-20-a-412010-r-30-a"
2023-05-15T10:53:07.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1842128.785585] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a-406000-r-30-a-419030-r-20-a-411010-r-90-a-421040-r-30-a-417010-r-20-a-412010-r-30-a-396010-r-20-a"
2023-05-15T10:55:39.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: stahtd: stahtd[2013]: [STA-TRACKER].stahtd_dump_event(): {""message_type"":""STA_ASSOC_TRACKER"",""mac"":""@@MAC@@"",""vap"":""ath0"",""event_type"":""fixup"",""assoc_status"":""0"",""event_id"":""1"",""arp_reply_gw_seen"":""yes"",""dns_resp_seen"":""yes"",""avg_rssi"":""-65""}"
2023-05-15T11:00:08.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1842549.824767] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a-406000-r-30-a-419030-r-20-a-411010-r-90-a-421040-r-30-a-417010-r-20-a-412010-r-30-a-396010-r-20-a-421020-r-30-a"
2023-05-15T11:01:38.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/2 msg of Group Key Handshake"
2023-05-15T11:01:39.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: group key handshake completed (RSN)"
2023-05-15T11:01:39.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: received EAPOL-Key frame (2/2 Group)"
2023-05-15T11:01:39.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: hostapd[2015]: ath0: STA @@MAC@@ WPA: sending 1/2 msg of Group Key Handshake"
2023-05-15T11:07:06.000-04:00,UpstairsWest,"UpstairsWest f09fc22cc9ec,UAP-AC-Pro-Gen2-6.5.28+14491: kernel: [1842967.864298] [DHCP-SM] @@MAC@@: r-40-a-450030-r-30-a-406000-r-30-a-419030-r-20-a-411010-r-90-a-421040-r-30-a-417010-r-20-a-412010-r-30-a-396010-r-20-a-421020-r-30-a-418000-r-40-a"

Jumbo Frames is not applicable on Wifi so luckily you do not have to worry about that. If you wired the hub to the LAN it would need to be separated from the JF or it will crash the Ethernet interface guaranteed. Its a known issue with the hub, possibly a hardware issue, not sure.

Does your Router/AP do any sort of scheduled channel "optimization" where it scans and adjusts the channel? This has been known to cause the hub to drop and not reconnect. I do not think they have fixed that issue yet. As a test try setting fixed channels and see if that fixes it.

Thanks! I've been having seemingly random (but somewhat consistent) crashes/lockups on my C8 myself. I did have jumbo frames enabled, but I moved it to wifi so that shouldn't be an issue any longer. It did still drop every couple of days (sometimes once a day), but your comment prompted me to check my connection logs and sure enough, when it did drop off wifi, it was at some point shortly after 3am. I don't know exactly what time my Unifi gear does its channel optimization, but it may be at that point (still digging through logs). I've turned that off for now so here's hoping that fixes it. Otherwise I'm wondering if it's some sort of process which runs on the hubitat itself?

As a side note, though, wifi really should try and auto-reconnect if it drops off or changes channel. Every device should do that, so not sure why this is a problem for the hubitat.