[Staff Support Request] Hub Networking Errors

[josh]

Cross-linking to the original post from a few days ago from the other more generic Get Help category in case there's any clues from there that might help others.

Hub network issues since last few updates

I have a C5 running 2.3.5.121. I'm all of a sudden having trouble with my hub and cloud-based apps. Multiple apps can't seem to connect online so I don't think it's an issue with any apps: [Screenshot 2023-04-03 at 7.51.15 AM] The hub Network Test all passes though without trouble. All of my non-smart home devices seem able to connect to the internet just fine, so I don;'t think it's a problem with my actual network. I have a Bark internet filtering device, but Ignore the Hubitat hub (and it…

@mavrrick58

...It looks like the majority of those are Sharptools. Can you try disabling sharptools to see if that will help as well.

SharpTools by it's nature is likely processing a lot more events than other apps, though with very little processing as the app is a lightweight event forwarder. Since most people who use SharpTools have several dashboards with many (most?) of their devices displayed in some fashion, the app ends up touching a lot of events.

That being said, it's a perfectly reasonable approach to disable apps one-by-one and see if anything causes the errors to stop.

[mavrrick58]

@josh

SharpTools by it's nature is likely processing a lot more events than other apps, though with very little processing as the app is a lightweight event forwarder.

Right, but something is consuming resources. The fact there is a out of memory error in that screen shot int he first post means something is causing a serious problem. I have only seen that error message pop up when there is a major problem somewhere in the hub with resources.

I use Action tiles myself and that app is similar, it doesn't do anyting but forward the events to their outside service. That said that application is consistently one of the busiest apps on my hub.

I would still like to know what the free memory is from the Hubinfo driver. My guess is we will find that memory is being consumed by something.

[JustinL]

No errors today, which I guess is good, but I suspect they will recur at some point soon. At the moment, hub info driver shows free memory at 198048

[jtp10181]

190Mb after 2 days is very low. Takes over a week for my hub to get below 200Mb.

Is the the V2 or new V3 Hub info driver? What is your refresh interval?
Also, what is the 3D Printer device? Is that a integrated via Wifi?

I would disable that 3D printer device and see if that helps.

[JustinL]

@jtp10181

Is the the V2 or new V3 Hub info driver?

v2.7.22

Polling rate was the 300 default but I increased to 1200 once I saw it was consuming a lot of resources

@jtp10181

what is the 3D Printer device? Is that a integrated via Wifi?

Octoprint cloud integration. I will increase the idle polling from 5 minutes to 1 hour to see if that makes a difference

[jtp10181]

@JustinL

v2.7.22

Polling rate was the 300 default but I increased to 1200 once I saw it was consuming a lot of resources

You should update to v3, it has 4 separate polling schedules so you can get the metrics you want regularly and the stuff that rarely changes you push out to update less often. [RELEASE] Hub Information Driver v3

May also want to reboot now, since memory is already very low. Start fresh with the changes and see how quick it drops.

[JustinL]

what's a good polling interval for free memory?

[jtp10181]

Here is how mine are setup, I am doing free mem every 12 minutes. I tried to make it so the two most frequent polls would not typically land on top of each other. Also there is no point in going below 5 mins for CPU since the hub only updates the metric every 5 minutes. Some people are doing shorter for mem but I personally don't see a reason for that.

[mavrrick58]

The reason for a shorter interval for free mem is to get visability to shorter interval fluctuations. I have seen occasions were my mem drops dramatically and then recovers within a few min. A 5 min or 10 min interval wouldn't show if that was the cause for a system wvent. I do one min interval, but use node red to collect it.

[JustinL]

New errors appeared today:

Device 1641 is flume
Device 1438 is Meater probe
The Sharptools Bad Gateway error is a new one though..

Hub Memory:

Does this rule out memory as being the culprit?

[mavrrick58]

What is your interval for collecting the memory?

[JustinL]

@mavrrick58

What is your interval for collecting the memory?

12 minutes (I copied your settings in the hub info driver)

[mavrrick58]

Can you set it to a shorter variable like 5 or 2 or even 1 min. That value is gathered as a instant value and can be gathered much more frequently. Once we get to the bottom of this then i would change it back to 5 or something larger, but having it set lower we could potentially get detail as to how long the memory is low or if it even goes lower.

It may also be good the next time this happens to collect and post the App and devices Stats page from the live logging info page. It may help atleast point us in a direction. All of those errors you are posting point to a networking issues like you are calling out. Are most of your devices wifi devices that use polling to obtain device status and such?

[JustinL]

@mavrrick58

Are most of your devices wifi devices that use polling to obtain device status and such?

Yes

[mavrrick58]

It may also help to see what functions where happening right before those events in your logs occured. Can you can the logs to see what activity was occuring just before those actions maybe that can show a precoursour event

What polling interval is your ecobee suite integration running at?

Also what time do you have the hub backup job running.

[JustinL]

New errors (happening a lot now it seems)

I had updated the memory poll to be every 1 minute, and it seems memory never drops very low

Ecobee polling is set to 5 minutes.
Hub backup set to 2:15AM every day

Latest device and app stats

[mavrrick58]

Can you adjust the Ecobee suite to 1 min instead of 5. The developer has indicated setting it lower is actually better with how it works.

On your hub memory graphic can you zoom in to only show the data that is by 1 min intervals.

Those Sharptools errors look like they are being triggered by the remote side. Do you have a firewall. What is your network setup like devices and how they are connected.

[mavrrick58]

by chance do you have a computer or other device on your network you could run a continous ping to events.shartptools.io to see if you are loosing connection to that host. preferably it would be on the same network and such as the hub so it can be as similar as possible in it's connectivity.

[JustinL]

@mavrrick58

Can you adjust the Ecobee suite to 1 min instead of 5.

Done

@mavrrick58

On your hub memory graphic can you zoom in to only show the data that is by 1 min intervals.

No, but here's the raw data:

Summary

HubInformation;FreeMemory;2023-04-1312: 08: 02.060;236024;Final,
HubInformation;FreeMemory;2023-04-1312: 13: 02.024;236024;Final,
HubInformation;FreeMemory;2023-04-1312: 18: 02.064;232596;Final,
HubInformation;FreeMemory;2023-04-1312: 25: 56.065;214608;Final,
HubInformation;FreeMemory;2023-04-1312: 26: 56.185;213052;Final,
HubInformation;FreeMemory;2023-04-1312: 27: 56.168;234928;Final,
HubInformation;FreeMemory;2023-04-1312: 28: 56.216;223268;Final,
HubInformation;FreeMemory;2023-04-1312: 29: 56.607;220624;Final,
HubInformation;FreeMemory;2023-04-1312: 30: 56.314;236268;Final,
HubInformation;FreeMemory;2023-04-1312: 31: 56.374;233276;Final,
HubInformation;FreeMemory;2023-04-1312: 32: 56.426;233420;Final,
HubInformation;FreeMemory;2023-04-1312: 33: 56.549;227844;Final,
HubInformation;FreeMemory;2023-04-1312: 34: 56.541;225676;Final,
HubInformation;FreeMemory;2023-04-1312: 35: 56.599;222652;Final,
HubInformation;FreeMemory;2023-04-1312: 36: 56.932;221932;Final,
HubInformation;FreeMemory;2023-04-1312: 37: 56.687;221304;Final,
HubInformation;FreeMemory;2023-04-1312: 38: 56.705;226220;Final,
HubInformation;FreeMemory;2023-04-1312: 39: 56.775;225056;Final,
HubInformation;FreeMemory;2023-04-1312: 40: 56.799;222536;Final,
HubInformation;FreeMemory;2023-04-1312: 41: 56.891;220796;Final,
HubInformation;FreeMemory;2023-04-1312: 42: 56.940;220636;Final,
HubInformation;FreeMemory;2023-04-1312: 43: 56.935;228192;Final,
HubInformation;FreeMemory;2023-04-1312: 44: 57.028;224284;Final,
HubInformation;FreeMemory;2023-04-1312: 45: 57.042;235940;Final,
HubInformation;FreeMemory;2023-04-1312: 46: 57.127;233348;Final,
HubInformation;FreeMemory;2023-04-1312: 47: 57.169;230168;Final,
HubInformation;FreeMemory;2023-04-1312: 48: 57.214;228464;Final,
HubInformation;FreeMemory;2023-04-1312: 49: 57.926;224736;Final,
HubInformation;FreeMemory;2023-04-1312: 50: 57.399;222236;Final,
HubInformation;FreeMemory;2023-04-1312: 51: 57.449;220712;Final,
HubInformation;FreeMemory;2023-04-1312: 52: 57.523;218988;Final,
HubInformation;FreeMemory;2023-04-1312: 53: 57.501;229676;Final,
HubInformation;FreeMemory;2023-04-1312: 54: 57.583;225444;Final,
HubInformation;FreeMemory;2023-04-1312: 55: 57.633;222248;Final,
HubInformation;FreeMemory;2023-04-1312: 56: 57.635;219120;Final,
HubInformation;FreeMemory;2023-04-1312: 57: 57.728;218704;Final,
HubInformation;FreeMemory;2023-04-1312: 58: 57.752;231116;Final,
HubInformation;FreeMemory;2023-04-1312: 59: 57.832;226376;Final,
HubInformation;FreeMemory;2023-04-1313: 00: 57.923;235592;Final,
HubInformation;FreeMemory;2023-04-1313: 01: 57.931;232740;Final,
HubInformation;FreeMemory;2023-04-1313: 02: 57.987;229172;Final,
HubInformation;FreeMemory;2023-04-1313: 03: 57.977;231352;Final,
HubInformation;FreeMemory;2023-04-1313: 04: 58.028;228108;Final,
HubInformation;FreeMemory;2023-04-1313: 05: 58.154;223740;Final,
HubInformation;FreeMemory;2023-04-1313: 06: 58.173;222196;Final,
HubInformation;FreeMemory;2023-04-1313: 07: 58.264;221548;Final,
HubInformation;FreeMemory;2023-04-1313: 08: 58.304;226604;Final,
HubInformation;FreeMemory;2023-04-1313: 09: 58.354;223044;Final,
HubInformation;FreeMemory;2023-04-1313: 10: 58.606;219644;Final,
HubInformation;FreeMemory;2023-04-1313: 11: 58.452;218836;Final,
HubInformation;FreeMemory;2023-04-1313: 12: 58.452;218484;Final,
HubInformation;FreeMemory;2023-04-1313: 13: 58.545;213772;Final,
HubInformation;FreeMemory;2023-04-1313: 14: 58.547;224944;Final,
HubInformation;FreeMemory;2023-04-1313: 15: 58.665;236492;Final,
HubInformation;FreeMemory;2023-04-1313: 16: 58.683;232076;Final,
HubInformation;FreeMemory;2023-04-1313: 17: 58.710;227844;Final,
HubInformation;FreeMemory;2023-04-1313: 18: 58.729;222832;Final,
HubInformation;FreeMemory;2023-04-1313: 19: 58.826;226140;Final,
HubInformation;FreeMemory;2023-04-1313: 20: 58.825;224312;Final,
HubInformation;FreeMemory;2023-04-1313: 21: 58.867;221184;Final,
HubInformation;FreeMemory;2023-04-1313: 22: 58.913;221096;Final,
HubInformation;FreeMemory;2023-04-1313: 23: 58.962;217884;Final,
HubInformation;FreeMemory;2023-04-1313: 24: 59.005;232396;Final,
HubInformation;FreeMemory;2023-04-1313: 25: 59.059;225544;Final,
HubInformation;FreeMemory;2023-04-1313: 26: 59.143;222484;Final,
HubInformation;FreeMemory;2023-04-1313: 27: 59.171;221632;Final,
HubInformation;FreeMemory;2023-04-1313: 28: 59.236;217804;Final,
HubInformation;FreeMemory;2023-04-1313: 29: 59.230;236184;Final,
HubInformation;FreeMemory;2023-04-1313: 31: 59.329;232604;Final,
HubInformation;FreeMemory;2023-04-1313: 32: 59.373;226680;Final,
HubInformation;FreeMemory;2023-04-1313: 33: 59.419;222840;Final,
HubInformation;FreeMemory;2023-04-1313: 34: 59.503;234904;Final,
HubInformation;FreeMemory;2023-04-1313: 35: 59.511;228620;Final,
HubInformation;FreeMemory;2023-04-1313: 36: 59.612;226880;Final,
HubInformation;FreeMemory;2023-04-1313: 37: 59.665;226200;Final,
HubInformation;FreeMemory;2023-04-1313: 38: 59.653;217740;Final,
HubInformation;FreeMemory;2023-04-1313: 39: 59.703;236608;Final,
HubInformation;FreeMemory;2023-04-1313: 40: 59.801;230240;Final,
HubInformation;FreeMemory;2023-04-1313: 41: 59.798;227636;Final,
HubInformation;FreeMemory;2023-04-1313: 42: 59.882;226388;Final,
HubInformation;FreeMemory;2023-04-1313: 43: 59.909;224204;Final,
HubInformation;FreeMemory;2023-04-1313: 44: 59.939;236364;Final,
HubInformation;FreeMemory;2023-04-1313: 46: 00.049;236916;Final,
HubInformation;FreeMemory;2023-04-1313: 47: 00.016;232060;Final,
HubInformation;FreeMemory;2023-04-1313: 48: 00.072;229936;Final,
HubInformation;FreeMemory;2023-04-1313: 49: 00.126;223740;Final,
HubInformation;FreeMemory;2023-04-1313: 50: 00.209;237204;Final,
HubInformation;FreeMemory;2023-04-1313: 51: 00.283;232248;Final,
HubInformation;FreeMemory;2023-04-1313: 52: 00.375;230128;Final,
HubInformation;FreeMemory;2023-04-1313: 53: 00.455;228920;Final,
HubInformation;FreeMemory;2023-04-1313: 54: 00.535;212544;Final

@mavrrick58

Do you have a firewall. What is your network setup like devices and how they are connected.

Yes, I believe my Orbi Router (RBR50) has a built-in firewall. My Hubitat hub has a wired (Ethernet) connection via a switch (has been for years and there haven't been any recent network changes).

@mavrrick58

y chance do you have a computer or other device on your network you could run a continous ping to events.shartptools.io to see if you are loosing connection to that host.

Yes, not sure how long to let it run for, but it seems I've gotten continuous replies....

[mavrrick58]

Another question. It looks like you have several Wifi devices and are filtering the logging to only show errors. Do you see any cloud integrations that are successful when these other ones are failing..