C7 hub unresponsive ever day or so - fix yet?

Though plugging the hub directly into the switch is good, it doesn't really apply to what I was saying. My point is that the hubitat hub is a small low power arm based SOC. It is fairly powerful for what it is, but it isn't as robust as lets say anything Intel/AMD based. When I had Node-Red on my unraid server reading a UPS dameon and then updating a virtual driver on Hubitat with many attributes i was able to kill the hub daily. It was because the APC Dameon pallet in node-Red had to refresh every 12 seconds. Then I was having it send the data to the hub for processing via the virtual driver for every attribute (10) the HE driver had. It seemed to work fine, but after 24-48 hours it would flake out and cause issues. The problem was completely about the amount of data i was sending it. after i reworked my node red flow to reduce the updates being sent everything was fine. Node Red actually interfaces with Maker API to load data back to the hub.

My question is really about how much continuous activity do you have with the services you are running. I don't know echo speaks, or the MyQ app, There is likely some kind of polling happening and how frequent is that.

In some cases infrequent polling can cause issues as is the case with Ecobeee Suite. With that application because it does so much, waiting to long to retrieve the changes from the cloud can cause it to have long runtimes on the hub and actually slow stuff down.

You may want to click on the Live Logging tab to display logging and then click on App stats. Then make sure everything is checked. That may be able to tell you if you have a application causing issues. You may also want to check this periodically as it may not be real good data after a restart. It also shows you total time and busy time so you can see in a way how busy the hub is.

You may also want to go into those apps and turn on debug logging as that may also help us understand what they are doing. what is shown up there is very minimal.

3 Likes

Thanks all for your feedback. After working with @aaron3 the problem may just be as simple as changing the Ethernet Speed in the Network Settings. In some cases, when the hub is plugged into a network switch that is able to auto-negotiate the link speed, keeping the hub on Fixed 100mbps could have unintended consequences. @aaron3 changed the setting to auto negotiated and also disabled the auto-rebooter, which may solve the problem. He will keep us posted.

5 Likes

I can second this observation and the proffered solution.

I have a LAN device updating 26 attributes via MQTT every minute that were passed on to a custom HE device via Node-RED. After a few days, the hub would get sluggish. Filtering the attributes with an RBE node solved the issue.

4 Likes

What you are describing is a software limitation (likely one or more bugs) -- the hardware could easily handle hundreds of queries a minute. MQTT packets are as small as 2 bytes so if you those 26 attributes are all separate queries and the JSON data (attribute data) is normal sizes (under 1k bytes) it should take under a second to complete all 26 requests.

Can you share what menu / app created the data on your screen shots? I've seen uptime in the past but for the life of me I can't find it now :grimacing:

@thebearmay's Hub Information Driver.

2 Likes

Absolutely agree +2

1 Like

I'm back... the C7 stopped responding to API calls and no longer interacted with Alexa... but today was the first time in 12 days it stopped responding to pings. Usually it doesn't last 12 days, usually 2 or 3 days. So the static IP may have helped but did not avoid the bug. Either the C7s web server is crashing or there is a bug in the IP stack or with the NIC.

Aug-28 6:35:17 PM Event Event Trigger "Hubitat Hubitat Offline - Write to Log"

yesterday I posted my last update then rebooted (un / re-plugged)... C7 lasted exactly 1 hour before it died.

Aug-28 10:34:42 PM Event Running script statement immediately: &hs.WriteLog(Hubitat, Offline)
Aug-28 9:44:46 PM Event Running script statement immediately: &hs.WriteLog(Hubitat, Online)

Any chance you can verfy if you have jumbo frames enabled, and if you do, can you disable? We've had reports of hubs freezing when jumbo frames are enabled.

When the C-7 becomes unresponsive, is it accessible from port 8081 (hub.IP:8081)? That would be a better way to reboot than unplug/plug, which can corrupt the Database.

Jumbo Frames are enabled on the entire network. I do not see a way to enable/disable on the C7. If C7 cannot handle JFs properly - that is a pretty big bug in the C7s IP stack or hardware. I'd hope you'd fix it. Any semi-modern IP stack in the last 10 years should be able to properly work on any network and negotiate frame size.

Asking people to disable JF on their network is not the answer. I have 50 network devices (~20 are wired) and none have issues with JFs... Rpi3 w/ custom Linux build, Yamaha AVR, Denon AVR, BeagleBone w/ custom OS runs my HVAC, NVidia Shield (Android TV), Sony PS4, Synology NAS, several PCs, and more all hardwired to my network run just fine.

1 Like

I am not network engineer by any means, but I am good at Googling. Even Neatgear warns about this:

"...the effort of configuring jumbo frames everywhere on your network is not worth the marginal improvement, and has the potential of slowing down or breaking non-jumbo frame clients."

The Ethernet port on C7 doesn't support gigabit speeds.

4 Likes

The biggest drawback when using jumbo frames is that they not defined by any Ethernet standard. Because of this each manufacturer plays by their own rules when implementing large frames. This has the potential to cause hard to troubleshoot issues. Unless every single device is configurable for jumbo MTU's, then you will run into problems and not all devices support it or are interoperable (this goes back to lack of defined standards) It is also not necessarily a bug that it can't handle jumbo frames, it's simply a case of not really being implemented. JF on a home network has no real benefit unless you are doing so much network processing that ever byte has to be at max speed, otherwise it's just a thing to do. I run a large network at home with 7 or 8 dell blade servers in the rack and 2 cisco 3750-x 48 port switches. I do a lot of processing and backup for clients and I wouldn't even bother turning on JF as it's really not needed. I mean take it as you will, but my point is that not supporting JF isn't a bug and what's the point unless you're running all the same brand supporting the exact same implementation. I've seen jumbo frames causing more problems than they're worth. Just saying.

6 Likes

Seems unfair to refer to lack of support for a non-IEEE standardized feature in a device with a fast Ethernet port, aimed at the home consumer market, as a bug.

I don’t think anyone wants to debate with you re: your own use of jumbo frames at home, or why you have chosen to implement it.

But it’s an edge use case that probably doesn’t impact many Hubitat users. So while it’s understandably frustrating, IMHO the staff shouldn’t be expected to drop everything and “fix” this.

7 Likes

You guys are chasing the wrong path. Lets be clear on the situation...

  1. Bobby did not say the problem is the C7 with Jumbo Frames. He wanted me to test. (Bobby, without messing with the network at all you should see the problem in the C7s IP stack debug logs - if those exist?)

  2. I am not saying the C7's problem is Jumbo Frames... no one actually knows.

  3. I have zero expectations (and never said) the C7 'should' support Jumbo Frames, not at all. Dont care. I said 'if' that is the problem, then there is a bug in the IP Stack since a proper IP stack negotiates speeds and frame sizes so no matter what, jumbo frames would never be an issue with a proper negotiation, period end of story. -- I have many examples of non-jumbo frame devices working 100% on my network. (provided above)

In the end, the C7 has a problem that none of the over 100 devices I've run on my network in the past 8 years has ever had... that I'm 100% sure of. This is the 2nd C7 unit that has the problem. Maybe I'm the only unlucky one that has the issue, I doubt it... its prob hidden by auto-reboots or restart due to constant use, or just constant use... it could be that a 99.9% idle C7 is the problem. Who knows.

I don't use the C7 for much, and I guess that is good since it does have a problem and I can't justify more time working on it. Maybe C8 will work better :slight_smile:

@gopher.ny, I thought the jumbo poison pill was being resolved over a year ago...

@aaron3, depending upon your switch, you can likely address your issue by adding an ACL to the Hubitat port to block outbound mDNS packets (UDP port 5353).

3 Likes

I don't know if jumbo frames are your problem or not.

But I can confirm on my (unifi based) network when I had jumbo frames turned on my multiple c7s would drop off the network fairly regularly. After turning it off they never do.

That is purely an anecdotal comment, and not necessarily what you are seeing.

8 Likes

Wrong. Yes there are reboots. But i just manually rebooted one of mine that had a warning of low memory. It was still working fine but had been up 33 days.

1 Like

@kahn-hubitat
Exactly how are my guesses (not facts) wrong? What does your statement prove? Just being up 33 days doesnt prove that there is not a network issue or that an idle system could not be a problem.

or maybe you don't understand I'm saying that no one knows what the issue is and it could even be something not network related causing it the C7 to become unresponsive.

1 Like