C7 Continues to stop responding

🛎️ Get Help
1

I have had my C7 for close to a year. I have discovered that when new code comes out, that i should not install it as it will consistently cause my C7 to stop responding to anything and stop processing automations. I have to pull the power to reboot it. I have opened ticket after ticket and i always end up getting "we dont see anything in the logs so are sending to engineering" and then never see anything else.

I had it running again where i didn't have to reboot it every day, and the latest code came out. I figured it was stable, and installed. Now it will stay active and responsive for an hour or less and then no longer respond to anything.

I have done a soft reset, factory reset, changed to previous versions of code, restored from backup and done everything i have seen in the help community, but to no avail.

This runs my security and safety lights at night and my pool alarm (windows and doors), as well as thermostats, lights inside and outside the house, so it has become a very large problem and a security concern.

I have it hardwired to ethernet switch. I have changed switches, ports, and cables. I am a systems architect for Cisco, so pretty clean network running including QoS to prioiritize network traffic to and from the C7.

Light stays green, have link access to switch, but nothing functions or responds, so no automation works (like turning on lights at night or activating motion sensors). The logs do not show any errors or issues, it just stops normal logging and picks up where i unplugged and replugged the power.

Is there anyone here or any engineering folks at hubitat who can help? I am ready to go back to home assistant because it stays up.

Thanks in advance for any help!

EDIT: JasonJoel, you may have had the key! Its been 4 hours since Jumbo Frames has been turned of (at 1544 right now), and things are running pretty clean and snappy. Will confirm tomorrow if it stays up! Thanks for the info!

EDIT 12-2-21: @JasonJoel gets the gold star! It has been rock solid and performing beautifully since i removed jumbo frames from that switch. Definitely something to look into code wise at some point. Most people would not have that issue so its a pretty small subset of users. I have another thing to try from @gopher.ny but gonna hold that for if anything happens in the future. I am calling this problem solved! Thanks everyone for the help!

3 Likes
2

Out of curiosity, do you have jumbo frames turned on on your switches?

I had a similar issue with one of my hubs on my unifi network, with the ultimate fix being to turn off jumbo frames on the switch that the hub is connected to (I probably could have just tried a smaller mtu, but turning off jumbo frames was easier).

For whatever reason the unifi switch was sending some weird packet sizes that the hub couldn't handle (even though it shouldn't have been).

3 Likes
3

I don't have much for you except what I learned yesterday when my system stopped transmitting all Z-Wave commands. My problem (so far) was resolved my removing 2 recently added Z-Wave devices. So, my first question is, is it only Z-Wave, only Zigbee, or both that are completely unresponsive? If one or the other, do they receive updates? Try a switch and see if it updates in the UI. Have you added anything recently (obvious question, I know, but in the end that seems to have been my problem.) I realize that's not much help, but we need to get started.

EDIT: I'm in way over my head, but it appears you're in good hands now.

4

Hi there, I took a look at your tickets and the last we received from you was back in May. Do you happen to have a more recent ticket that I can take a peek to see what is going on with your hub? If not, please send us a new email. In the ticket from May we suggested that some custom automation may have been causing the problem, and because we didn't hear back, we assumed that the problem was resolved.

1 Like
5

Wow. That’s odd. I’ve never seen that on my C-5 or C-7. Because you are a systems architect for Cisco, you clearly know what you are doing, so I’m going to make a wild guess and ask a few questions.

My wild guess is that you might have some nonstandard (for most of us) network ingress/egress filtering. I believe that one of the things the hub now does on reboot, just at the end, is to check subscriptions (“fetching entitlements”) and phone home to establish a NAT traversal tunnel for cloud backups and VPN, if you have subscribed to those.

Is it possible that the hub is hanging on this by some firewall rules you have in place?

Are you able to see your hub on the Hubitat portal (my.hubitat.com then log in then Find Hubs)? If, on the Hubitat portal, you go to Registered Hubs, then Hub Details, is the last checkin time current?

I’m scratching my head on this one, so it’s got to be something different about your setup as compared to mine.

Perhaps if you put a simple systemStart() rule with a log message (Trigger: Location Events, systemStart), that would show that an initial rule gets run.

2 Likes
6

I definitely would have what you would call non-standard ingress/egress :slight_smile: From the hubs perspective, its pretty clean, Its whitelisted with no bandwidth limits and is NAT'd directly out. I have zero issues seeing it from the hubitat portal (when its responding). I will have to check on the last check in time to see if there is something that corresponds to the times it just goes out to lunch.

1 Like
7

Can you PM me your hub's id?

1 Like
8

Great question! I DO have jumbo frames enabled on the network. Mostly because of the two NAS systems that share the same uplink to the core. They sit on an access switch, whereas i have the hub on the distribution switch to give it some relief during backup and syncs of the NAS boxes. I will change back to something closer to 1500 as an MTU and see if it helps. I will know within an hour...

9

Thanks for any and all input! To answer your questoin, its all Z-Wave with some API calls for the thermostats. When it works, it works solid. Its just a matter of having it stay up.

10

BobbyD, Thanks for reaching out. I do not have any mail from you or hubitat saying anything about a custom automation. The ticket i had open on this issue was 21819. I had another open ticket 21820, where my new C7 i bought to go with this one had bricked. I received a message from you at 3:54PM EST saying you closed the first ticket and merged with the bricked C7 ticket. I responded at 4:07PM and have not seen anything else from Hubitat or you.

This was the last mail i recieved: " Hello,

Thanks for taking the time to reach out to us, we're here to help! Having a green light is good. It likely means that the hub is working but cannot join the same network as the one that the device you are using to access the hub is connected to. I would try basic network troubleshooting.

Here are a few tips that may help resolve your problem:

  • Use an alternative device to attempt to connect to the hub (we recommend using a computer for the initial set-up)
  • Use an alternative browser (we recommend Google Chrome for the best experience)
  • Make sure your device is connected to the the same network segment (i.e. same subnet)
  • Reboot your router
  • Power cycle your hub by unplugging the power cord for 30 seconds before plugging it back.
  • Replace the Ethernet cable.

If your problems persist, please provide us with the following details, so we can further research:
" I did all the things you had requested :slightly_smiling_face:

11

Just wanted to be more clear. The C7 comes up and works. i can connect using a browser, it will do automation (if thats the time it is), i can turn things on and off, tiles reflect correct settings, etc. It just flat stops responding.

Here is a sample from the log of it working, and then you see a jump in time where i had to pull power and restart it:


dev:1592021-12-01 12:03:42.876 pm infoInitialize Poll

dev:1492021-12-01 12:03:42.617 pm debugNow: 1638378222616, JS length: 1015, Maximum is 1024,

dev:1492021-12-01 12:03:42.277 pm inforefresh()

dev:652021-12-01 12:03:42.273 pm traceInitialize virtual Hub device...

dev:1582021-12-01 12:03:42.261 pm debuginitialize...

dev:352021-12-01 12:03:42.250 pm infoInitialize Poll

app:702021-11-29 08:43:15.316 am traceReceived ping from Smart Things Hub.

app:702021-11-29 08:42:15.190 am traceReceived ping from Smart Things Hub.

app:82021-11-29 08:41:50.728 am debugdevice event: {"name":"humidity","value":"30","displayName":"Thermostat - Basement","deviceId":"68","descriptionText":"Thermostat - Basement humidity is 30 %","unit":"%","type":null,"data":"{}"}

app:702021-11-29 08:41:50.690 am infoReceived event from Smart Things Hub/Thermostat - Basement: [humidity, 30 %]

app:82021-11-29 08:41:36.213 am debugdevice event: {"name":"humidity","value":"35","displayName":"Thermostat - Living Room","deviceId":"70","descriptionText":"Thermostat - Living Room humidity is 35 %","unit":"%","type":null,"data":"{}"}

12

Yup, that's the last ticket. I sent you a follow email on May 7th at 5:42PM indicating that it may be a custom code. Sorry if you didn't get it. We will get you sorted out if you still have problems.

Please take up @gopher.ny offer above and he will take a look at your hub.

1 Like
13

Done

14

Thanks Bobby! Unfortunately, i never recieved that message :frowning:

I have responded to gopher.ny so hopefully we figure this out. I dont have a lot of hair left, and really want to keep the three remaining ones!

2 Likes
15

Don't let things get out of hands for so long. If you don't hear from us, fire back another ticket. That will get our attention. We value your hair, so we want you to cut it not pull it.

1 Like
16

LOL! Thanks! Its more falling out on its own :wink:

2 Likes
17

Did this ever get resolved for you? I'm running into a similar issue but I don't have the luxury of turning off Jumbo Frames.

18

I doubt it. There are many IoT and consumer devices that react poorly with jumbo frames, Heck, there are even some switches that do weird things with the fragmentation when connecting to devices that don't support jumbo frames too, even if the device itself is OK with the fragmentation (I'm looking at you Ubiquiti...).

I've personally seen dozens of consumer devices that will do wacky things when jumbo frames are turned on on the switch/network. That's why I only use jumbo frames on my isolated storage networks (where all devices and the switch support and are configured to use jumbo frames) now.

Moral of the story - Jumbo frames are great when you are going jumbo frame device to jumbo frame device, but unpredictable when you are mixing and matching jumbo frame with non-jumbo frame devices. That is why pretty much every network equipment manufacturer says don't use it unless ALL devices support (and are configured to use) jumbo frames. Weird stuff happens sometimes when the network devices have to start fragmenting the packets.

2 Likes
19

I am curious as to why that is?

20

Well I just was reminded my hub is in a separate vlan than my access points so I actually can turn off jumbos. All of my wireless traffic is tunneled, and that was why I needed jumbos. But just realized I moved APs to their own vlan recently.