Why does my new c8 hang?

Just starting out with my new C8 in a new house and everything has been working fine. I have only 5 devices (4 z-wave+, 1 zigbee) and 2 rules and run the hub wi-fi. My wi-fi is strong all over the house w/ no problems as is the z-wave (added them in place from the central wi-fi hub just fine). The C8 has hung twice in the last week where no devices responded yet the status light was green. I have to confess I did not try the web interface (should have) or the interface I read about but cannot remember [hub address]/debug or something? Powering the C8 off/on had everything working again, so some questions to help diagnose if it happens again (besides trying the web interface) -

  1. Has anyone experienced the C8 hanging and the status light was green?
  2. What is the alternate URL to get to the hub to learn more about status?
  3. any other ideas?

Thx, Dave

Don't power it down - that might cause database corruption.

There is a diagnostic menu -

  1. Navigate to http:// [Your Hub IP] :8081/ and you will see the Hubitat Diagnostic Tool main page.
  • NOTE: Replace [Your Hub IP] with the IP address of your hub. You must use the IP address; a hostname will not work.

More information at:

It sounds like you might be experiencing Z-Wave problems.

1 Like

What networking equipment do you have (brand)?
Have you ever enabled jumbo frames on any hardware?
If a jumbo frames packets hit the hub, it can cause it to crash. Could be from a single device on the LAN sending out a multi-cast.

Also what Eric said, dont just pull the power on it unless the UI and Diagnostics are both inaccessible. Even then you could do a network reset (hidden button underneath) to do a reboot.

2 Likes

Sorry I had to jump off - I had an incoming phone call.

Check out your Z-Wave Table at Settings > Z-Wave Details.

Look for any devices with a "Discover" button in the Status column - that would indicate an incomplete joining of a device. Also look at the Security column - you would like to not have any devices with S0 security. Then look at the Route column - you are looking for any devices that have no entry here. That's not necessarily indicative of a problem because it takes a while for this column to populate (a day or more is not surprising, especially for battery powered devices). If you see no route, test them from their Device page to see if they respond.

The Discover button, indicative of an incomplete join, indicated you have a ghost or zombie. That is an entry in the database but there is no active device there. If you see any of those - do a quick search on Z-Wave Ghosts - there are a lot of threads on dealing with them. They are a serious problem because the system may try to use them to route instructions and due to the incomplete joining they aren't functional - this then disrupts the traffic to the downstream devices.

1 Like

I inherited a Ubiquiti network when I bought my new (to me) home recently. The US-24-250W Core device is the only device I see w/ a "jumbo frames" setting which is not set. Don't see any "jumbo frame" or MTU option for the router (USG-3P), Cloud Key (UCK-G2-Plus) or any of the other Ubiquiti devices. Supposedly can be set via SSH but was not by me if it was so set. Only other devices are my Synology server, Windows laptops, Android phones, Sonos speakers, Firesticks, Alexas, Kindles...I'll try to contact the web interface and diagnostics if this happens again. Thx!

All Z-Wave device status show OK, no "Discover" or "S0" Security and two showed routes, one did not (there were actually only 3 z-wave+ and 2 zigbee devices, not the 4 and 1 like I originally said). The Fan Switch that showed no route responded via the device page, the actual fan switch, and my Echo device just fine. I had never done a "repair" since adding the devices, so I did that successfully w/ no apparent changes to the routes. All status still Ok and fan switch route still blank.

If 01 is the hub, then all routes seem direct to the devices that show a route. Topology matrix shows blue for all legit combinations but red for the one device that had a blank route which is the fan switch that is working.

Fwiw, when the devices quit working, the zigbee stuff did not respond either.

Thanks for the diagnostic URL and info page and the advice to not just cycle power.

Apparently you have to enter the hard wire MAC to login and not the wi-fi MAC being used? My network showed the wi-fi MAC of the hub but Hub Details showed the hard MAC. The wi-fi MAC did not work to login to the diagnostic port but the hard MAC which I was not using did work to login. Both IPs are fixed in the network.

Also decided to update from version 2.3.5.105 to 2.3.5.113 for good measure.

I have the same problem. I was on 2.3.5.113 and I'm rolling back to 2.3.5.110 to see if that would stabilize the issue. I have about 130 devices. 4 devices have S0 (Yale door locks). I'm not clear on the issue with S0 devices. Is there a bug I should be aware of. I also am very care with ghosts. I have none and I'm quick to clear them out if I get a failed add.

Update. The rollback didn't help. I did a little digging and I see inbound Z Wave actions from my scene buttons coming in and I see the logs are trying to execute on those actions but the actions never get sent to the Z Wave Radio. I also did a manual test to see if I could turn on / off a single device and nothing is now going out the Z Wave Radio. Inbound only seems to be working. When I power down the Hubitat and pull the power on it then cool reboot, I'm back to normal operation for about 12 hours.

S0 is a security method that uses 3X the traffic as other Z-wave methods. Used when not needed, it can bring a network to its knees.

Barrier devices like locks and garage doors require use of security S0 or the much improved and less harmful S2 Authenticated method. And that makes sense.

But some devices require S0 on things like power reporting outlets or multi-sensors that are already very chatty. Adding the S0 triple traffic on top of a device that is sending lots and lots of traffic is what causes the majority of issues. Avoid pairing devices like this with S0 at all costs.

2 Likes

Things were fine last night and broken this morning. I cannot connect to the hub website or to the diagnostic port. Hubitat still green. Unifi shows my C8 hub disconnected around 3AM this morning. Also shows some Hubitat disconnects and connects earlier during that evening and two regular occurring IP conflicts, one of which is one of my two fixed IP Synology server ethernet ports and the other, I'm not sure as nothing is using that IP at all. Log suggests a possible rogue DHCP server but don't know where I'd have that. Also shows one of my more remote Sonos speakers often disconnecting/reconnecting while claiming the client experience is excellent and shows my APs switching channels each morning at the same time my Hubitat disconnected and did not reconnect. The APs are set to optimize channels once per day and seem to switch twice every morning around 3AM. This is getting beyond my network experience given I'm a C# et al programmer and not a network engineer...I'm going to try to research/monitor the IP conflicts and look into maybe turning off the AP channel optimization. Think I don't have any option here for the Hubitat other than cycling power. Any ideas/suggestions appreciated. Thx

Does your Hubitat lock up and you cannot connect normal or via the diagnostic port?

This really seems like something throwing out jumbo frames on the network. Install wireshark, then mirror the port that the hubitat is plugged into and to a cap. When it locks up go back to when things were normal and you should see the packets hitting the hub and from where.

2 Likes

Rick, What do you mean by "...and to a cap"? Hubitat is wireless and not plugged into an ethernet port. Can I still do what you suggest?

If you're using an AP to connected to the dream machine you can mirror the port that the AP is connected to.

Cap=capture as in Packet Capture (capturing the data to a file). You want to use WireShark for this. Filter for MTU above 1500 when examining the captures that are generated.

Good question, the GUI appears to remain working and I was able to roll the firmware back to 2.3.4.148 and things seem to be stable today. If it lasts until tomorrow then I suspect It's something to do with 2.3.5.x firmwares. It's as if the z-wave radio gives up sending and only receives. Very odd. Oh, note, while downgrading from 2.3.5.13 to 2.3.5.10 to 2.3.4.148, in that order, each time I did not hard cycle the C8, optioning to let it reboot itself. When I finally hard cycled the C8 on the final 2.3.4.148 revision after it too held on to this odd won't send z wave commands issue, things again returned to normal. That was last night and typically this issue will come back within 24 hours so I'll know more tomorrow.

question, will jumbo frames crash the Hubitat? I have a UI Dreammachine as well, and I have jumbo frames turned on so I can use my NAS at max throughput for large files. Is this a known issue?

1 Like

Yes, Jumbo frames will bring down the hubitat. It is a very well known issue. If you can isolate the Hubitat on it's own vlan and block jumbo frames from that vlan, that will solve it

question, is a fix going to be created resolving that issue? That sounds like a serious bug. When we say down? does the system free up and become unresponsive. That doesn't seem to be my issue and I've never had jumbo frames impact my C7. While the my C7 wasn't perfect and sometimes very slow to respond to automation, it never crashed or locked up.

Not unless the ethernet card changes which I doubt. It's been a known issue since it's inception several years ago. It's a hardware characteristic of the chip.

Rick, I've downloaded and installed Wireshark on my Win11 machine. Trying to do what you suggest, at the welcome screen, I select Wi-Fi under Capture and set "...using this filter" to "greater 1500". Is that going to capture jumbo packets?

What do you mean by "mirroring the port"? I see where I can mirror the AP port to another switch port but how does Wireshark watch just that port rather than wi-fi in general?

Also, my Ubiquiti switch does not have the Jumbo Frames option set, so could I still be getting jumbo frames given all traffic goes through that switch?

My NAS server interfaces have both MTU set to 1500.

Thx

You plug your laptop into the mirrored port and capture everything going through it

1 Like