C-8 inexplicably went offline this morning

I haven't made any changes to my hub in the last few days, and it's been very reliable for months.

Problems:

  • C8 stopped working with zigbee, zwave, cloud, and local integrations.
  • LED appeared green.
  • Ping would respond about once every 60 seconds, couldn't hit the main page, nor the diagnostics.
  • Watching the indicator lights on the switch side, I could tell the hub was cycling the network interface every 2-3 seconds. Likely responsible for getting random pings, but no real responses from the UI.
  • After many reboots, trying different USB PSUs, it was starting to do the red/green error. My Unifi enterprise switch has never had an issue, and I have POE disabled for this port. I tried moving it to another port, to no avail. Tried new cables, etc.
  • Attempted network reset -- knowing it probably wouldn't do much for wired/dhcp setup. Pings started to respond every 15 seconds, which was a nice improvement but doesn't fix anything really.
  • I unplugged the Ethernet completely, did another network reset, and put the hub on my Wi-Fi. The main page is still dead. I did notice if I let the page refresh enough times, I DO see that the URL redirected to http://192.168.75.31/brokenDatabase (title "Corrupt Database,") which is odd. What causes a corrupt DB, and why would that cause the Ethernet to cycle like that? Seems weird that I was able to get the Wi-Fi online. Seems like some sort of bug causing the other interface to cycle.

I'm going to perform the various resets, and restore the backup if necessary. The whole thing seems odd.

Often users don't realize that changes are made on their behalf every second or less without human interaction. The hub may be subjected to a brief power outage, or network outage, a router may be (auto)updated and could result in effectively blocking hub's access to the network. Plus devices connected to the hub make changes, and when some misbehave they could negatively impact the hub's performance.

Corrupt Database can occur after a brief power outage, or if one or more devices are misbehaving for a prolonged period of time.

1 Like

To resolve a corrupt database issue, you will need to perform a Soft Reset by following instructions in below document:

https://docs2.hubitat.com/en/user-interface/advanced-features/soft-reset

1 Like

Blinking red/green means that the hub lost Wi-Fi connectivity. Perhaps your Ethernet connection was down and you had Wi-Fi enabled, so the hub was trying to connect via Wi-Fi. Is the hub now connected to the Ethernet and has a steady green light? Can you reach the Diagnostic Tool?

1 Like

No matter what amount of tinkering I tried, I could not get the hub to use the Ethernet. Despite network resets, reboots, etc, I could only ever get random pings evert 15-30 seconds. Couldn't load the diag page, or even get a partial response via curl.

I did not attempt to manually set the speed of my switch, but that is just bonkers, when everything else is working fine.

My Wi-Fi was configured, but disabled previously, IIRC, they didn't work well together at the same time. (Been a while since I touched that side, since the Ethernet portion has worked so well.)

When I say I didn't make any configuration changes, I mean I didn't modify anything. I'm aware that devices are generating logs, and of course I've got a lot more advanced shenanigans going on than the typical user.

I'm mostly just surprised that the physical Ethernet connection kept cycling. I did the soft reset, restored latest, and now the hub is functioning on Wi-Fi.

I'm about to go back in there and see if Ethernet is once again happy. That would be the weirdest part for me, is if the corrupt DB is what was causing the physical port to cycle.

Is there any code in the hub that intentionally down/up the port for basic troubleshooting? Just curious if it could find itself in a runaway situation.

The hub was looking for a network connection, which means the Ethernet connection was likely down. The hub only uses one interface or the other. Also make sure that you are running the latest platform version, as the latest update has hub connectivity enhancements.

Interesting development. While the port(s) were not down-down and all the ports work just fine with other equipment (on different VLANs,) for whatever reason it seems like the C8 can no longer stay connected to my Unifi USW-Enterprise-8-PoE. Neither can any other device that uses the same VLAN. Once I move it/them to my UDM SE, the Ethernet light is static.

About the only thing left I can think of is is to perform a factory reset of the switch. Just a bizarre problem to crop up while sleeping, and no other changes. I'll let you know what I can figure out on the switch side.

Sounds like Unifi slipped in some rogue code. I cannot explain why else one specific VLAN would fail to route properly.

My guess is that between the Ethernet-bouncing, and the power-cycling during troubleshooting, that's why I ended up with a corrupt database.

Just in case anyone else experiences this issue -- despite not being the fault of the C-8.

I've discovered the 'bug' on the Unifi side. Something they modified in their code broke the way ports are assigned to certain VLANs. Normally, I just tag a port with VLAN X, and tell it to block all other traffic, (or auto if I'm trunking others.)

What appears to have corrected the issue is in the DHCP assignment page for each entity, you can not only set the static IP, but you can force a virtual network override. This should happen naturally based on the address assigned, but for some reason this functionality broke last night/this morning without touching a thing. Setting this to force the correct network allowed me to re-cable everything back like it was, and everything is happy.

I'll leave the Wi-Fi enabled as a nice backdoor in case something wonky ever happens again. Thanks guys for helping explain what the LED/port indicators should be showing.

Have a good one.

2 Likes