When the hub stops responding:
- What’s the color of hub’s LED?
- Can you ping the hub’s IP?
- Can you reach the hub’s Diagnostic Tool (port 8081)?
I'm using the built-in Hue integration. Only built-in apps, nothing custom or from HPM or Github. Just a stock C8.
Green, no, no.
The symptoms you describe are what I would expect if the hub's network interface received a multicast jumbo ethernet frame.
Can the ethernet switch the hub is connected to be configured such that the hub's port is restricted to 100 mbps. That'll prevent jumbo frames from being sent to anything connected to that port.
Alternatively, do you still have an old Fast Ethernet (i.e. 100 Mbit) switch? Connect that to your network, and connect the hub to the Fast Ethernet Switch. That's a simple way of ensuring that the hub doesn't see any jumbo frames.
I had this issue several years ago, and went through all my ethernet devices and configured them individually to not use jumbo frames. Now I've placed all my iot hubs on a separate 100 mbit switch.
1 Like
The hub is connected via wifi. There should not be any jumbo frames on my network. Jumbo frames are disabled in the router and all network devices have default MTU's.
I disabled Homekit integration. I'll let it run for a week or so and see what happens.
2 Likes
The next time this happens, can you try plugging a cable into the ethernet port to check if the hub is reachable? Just to test if the WiFi interface is going down for some reason.
Also, when the hub is unreachable, do automations still run?
No, automations do not run, sensors do not respond, the hub is fully locked up.
@JoeP - I have a similar setup as you, except my hub is hardwired.
One suggestion I have is to PM the support_team with your Hub ID and exactly when the lock-up occurred. They tried looking into my issue and were unable to because the logs were insufficient and therefore didn't have enough to go off of. I've learned they have access to certain data we don't they refer to as the 'engineering logs.'
2 Likes
That is definitely not a network issue. And I don't think that either the Hue integration or the HomeKit integration should cause a lock up like you're describing. This could be a hardware issue, and therefore I think it would be useful for Hubitat support to take a look at the engineering logs on your C-8. I am tagging them below:
2 Likes
The hub locked up again last night. All native apps, only the Hue integration running. I'll PM support.
Hue integration is a lan app.
I have had a few weeks of No Lockups. I am still here following this. When (if) it happens again I will provide requested diagnostic tool info for support.
2 Likes
I'm having frustrating lock ups here. I've scaled down all devices, apps, automations. Basically "barebone" to see if this continues. I'm using a scheduled nightly reboot as well to help keep things fresh. Nothing out of the ordinary on the logs - apps and devices specifically.
By chance do you have jumbo frames on your network.
That certainly isn't normal and you may want to open a support ticket.
2 Likes
Been experiencing the same issue described here; new C-8, becomes completely unresponsive, diagnostic tool's unreachable, web UI's unreachable, no response to ICMP pings, router doesn't detect the device, switch doesn't detect anything is connected in the UniFi controller, but does detect that the port's connected and drawing PoE power (but no network traffic). Unfortunately I cannot test anything more "direct" as I don't currently have any Hub-managed Z-Wave/Zigbee devices controlling one another. I have no integrations or apps configured other than the native Hue app.
Hub is hardwired via a PoE splitter to a UniFi USW-Pro-24-PoE. Port was set to autonegotiation, but only negotiated full duplex Fast Ethernet. Just went through and set the switch port to skip negotiation and only do FE FDX. Also manually disabled jumbo frames on all switches, though everything on the network (both infra and clients) have MTUs set to 1500 and nothing should have been using jumbo frames. I can't imagine what could've generated a multicast packet big enough to go into jumbo frame territory.
Since I just made the port speed and jumbo frames configuration changes, and this issue happens only once every/every other week, I don't know if they've solved anything, but I'll report back if I notice another lockup.
FWIW, I migrated from a C-7 which had zero issues, it was stable for months on end with even more integrations configured (Chromecast, Google Home, and a community version of the Google Home integration). The C-8 has been doing this from the get-go, even before I ran the migration; didn't have time to run the migration and test everything for a few weeks after hooking up the C-8 on the network, and multiple times when I tried to bring up the web UI to start the migration it was unresponsive and had to be power-cycled. That was on an older software version, too, which I had to upgrade from in order to get the web UI to even show the migration option (and for some reason didn't use the DHCP-provided DNS servers when told to check for updates).
Known issue using POE splitters and crashing the hub.
That is also a known issue that will crash the hub.
Playing with the DNS and Static IP in the hub settings is very easy to misconfigure. I am not seeing whether you were setting an address reservation in the router or not, but you should be doing it there and setting DHCP in the Hub, and not doing anything with the DNS stuff in the hub.
3 Likes
Can you PM me the hub id? I can check the engineering logs.
I appreciate the response. Please let me clarify:
- I'm aware from the comments on this thread and others that there's some known-issue involving jumbo frames, which is why I went through and disabled that feature on all of my switches. It was enabled, but I don't have anything that's using jumbo frames, so I disabled it just to rule that out as a possible issue.
- If there's specific PoE splitter models that are known to cause problems, that information would be appreciated.
- Regarding DNS, the issue isn't anything mentioned here. The issue was that on the default platform version the C-8 shipped with, it would not use the DNS servers it was provided via my router's DHCP server. I don't recall the exact details, but it either sent no DNS queries out at all (verified via pcap taken from the switch), or was trying to go straight to Google Public DNS. This resulted in the update check failing due to "DNS timeout" and the version it was on has a known issue with the migration feature not being available and/or not working.
- The DNS issue was solved, and is no longer relevant after successfully upgrading the hub firmware. To correct it, I went through the first-time setup wizard and configured the hub as a new device, changed the DNS servers under Settings > Network Setup, and afterwards it was able to successfully perform the upgrade. I then reset the hub back to defaults, and went through the FTSW again and selected to perform a migration, which was successful.
- The hub is not currently configured with manual DNS, nor is it configured for a static IP, it's getting everything through DHCP and is using the DHCP-supplied DNS servers (my two Pi-hole servers). A DHCP reservation is configured on my router, and there are zero issues there.
tl;dr, disabled jumbo frames to rule that out despite a lack of evidence confirming it to be the root cause. DNS and DHCP is not an issue after the platform was upgraded and the hub was reset several months ago, and is irrelevant to the current issue of the hub being unresponsive. All network settings on the hub are set to default and have been since it was setup.
Done!
Side question for the Hubitat team: Is there a way to access lower-level logs from the hub as an owner? For example, accessing and pulling the equivalent of /var/log/messages via SSH? Or having the hub output logs via syslog, since the web UI and logs contained therein are unavailable when the hub locks up, and upon restart the hub logs are empty? Having more serviceability/self-service options is always appreciated!
Interesting. My lockup issue was “solved” by switching to a wired connection instead of Wi-Fi. It has been several weeks now without a lockup. Of course the Wi-Fi is a feature that I want and paid for so I expect that Hubitat will resolve the issue.
No it just seems any poe splitter+unifi+c8 is a bad combo. C7's are fine.. Mostly it causes z-wave issue which cause the lock ups. Seems to be some kind of RF thing, unsure.
Also with all your power cycles from lock ups, I would do a soft restore to ensure a clean database.
1 Like
