Easy test would be to roll back? I thought as you did but then when I rolled back it was immediately obvious.
I suspect it's been too long... and I don't currently copy down backups, so probably a good prompt for me to set that up.
I'm happy to work with 148 for the moment, though getting rather frustrating at times. I'd like to use it as an exercise in troubleshooting if nothing else.
1 Like
Actually port 8081 keeps the last 3 versions, so you may be able to roll back to at least 135.
3 Likes
Great tip, I had no idea, I can even go back to 129 if I wanted.
I might not use it myself this time, but will certainly keep that in mind.
I think it would be useful if that was made known to users as they applied an upgrade, that and an option to download the pre-upgrade backup outside of the hub. Perhaps even nominating / configuring a cloud hosted platform of choice, e.g. Google, OneDrive, etc.
1 Like
I believe there were quite a few changes in the 2.2.3.x platform to enhance/improve overall platform stability and performance. These changes may have sacrificed the admin web interface performance slightly, as compared to 2.2.2.x. However, my hubs have been very stable on 2.2.3.
1 Like
when you see slowness (on later releases) check these endpoints
http://hubIP/hub/advanced/freeOSMemoryStatus
and
http://hubIP/hub/advanced/freeOSMemory
This will give you information if low on memory on the system. Will let you try to correlate the issue. There is other debugging that can be done...see
4 Likes
I get Green and 360740 respectively on the links.
Is that OK?
yes - says memory is good. So you do not have a memory shortage.
The correlation of reported free OS memory to hub performance may be a little nebulous, at least in my experience. When I tracked free OS memory on 2.2.3.142, after 150 hours uptime it showed 359K available yet the UI performed very sluggishly; apps begain failing and the hub required a reboot. Yet on 2.2.3.145, after running for 377 hours the hub reported 255K free and still ran acceptably well (though it eventually required restarting a couple of days later).
This leads me to wonder if some kind of thread synchronization is the root of this issue rather than a memory leak (like what percentage of time the CPU is spinning on locks vs. doing useful work).
so the next test would be do you have 'runaway process' eating the cpu cores.
In the wiki above, there is enableStats and disableStats, so you may want to enable for a while and see if something is running too much (eating the CPU AND HEATING UP THE CPU).
On C4s, do monitor your temperature - they are more likely to turn off cpu cores due to temperature issues than C5, C7. They do attempt to turn the CPUs back on, but if it is hot, they will go off again. You can test this by increasing airflow around it for a few hours if you see the slowdown with a lot of memory. I don't think HE exposes via web hook cores online or cpu load average.
I keep my C4 vertical (vents exposed) and not in a too warm place...I have never had a temp issue with C5 or C7.
1 Like
I've noticed this as well. I've also been having other issues. I have gone to weekly reboots that I did not do before. It's gone completely unreachable twice in the past few weeks. I have no led's, stopped working a while ago. I didn't post as I haven't dug in at all and having nothing to add. I was unaware of the 8081 having the last 3 versions. Thanks @april.brandt and I will follow @nh.schottfam advice, thank you. I did notice my hub was unusually hot the couple of times it went unreachable but it's in the same spot as always and the ambient temps have been cooler.
1 Like
Count me in for this too on a C4. Woke up with unresponsive hub a couple of days ago. Last night the hub website did load at all.
In my case UI latency increased (by my subjective estimate) by up to a second, although I don't think rule/automation performance took a hit (at least not that I could discern).
I believe it's the last 3 updates you actually made to the hub rather than the actual released versions. So if you applied all 7 releases in 2.2.3 you wouldn't be able to go back to 2.2.2
Not as far as I know unless you have devices whose built in drivers didn't exist before or use functions that weren't available previously.
I wasn't aware of these hub functions (beyond the hub diagnostics page), thanks for bringing them up! I haven't done these tests as a rollback of the firmware with no other changes brought an immediate improvement in UI performance so unlikely to be a direct fault of devices/apps?
2 Likes
I only want to add a data point... not derail the discussion...
I have 7 hubs... a C-3, C-4, C-4, C-5, C-5, C-7, C-7 and they are all on v.2.2.3.148
The C-3 and both C-4's were upgraded to .148 on the 16th and that is their most recent restart:
One of the C-7's was upgraded to .148 on the 16th and it was rebooted on the 22nd, and I can't remember the reason.
The above 4 hubs are part of my Production set, interconnected via HubConnect.
I have a C5 and C-7 as my Development set, interconnected via HubConnect. Because of their role as development hubs, their restarts are usually related to soft resets I do to swap in various configs. The C-5 had 3 restarts while on .148 and the C-7 had 2.
My final C-5 is also development but not interconnected. It was upgraded to .148 on the 16th at the same time as the first 4. No restarts.
Again, my purpose for detailing this is to indicate that YES, the hubs shouldn't need rebooting, and it is correct to try and find ideas about how/why/when anomalies occur. The reason I say YES, is because all of my hubs work fine on .148 without restarts-for-slowness. Yours should too.
7 Likes
Thanks for that - do you have many devices paired to each hub? (I have only a C4 with about 35 devices). Also to reiterate I don't think I had any performance issues other than with the UI.
Ah and they're all zigbee (zwave too expensive in Australia).
My C-3 has 57 ZWave, 0 Zigbee, 98 devices total
My C-4 has 25, 23, 69
My C-4 has 0, 0, 160 (This is my HubConnect Server Hub with the radios disabled.)
My C-7 has 13, 0, 18
My C-5 has 1, 1, 9
My C-7 has 2, 0, 10
I have 143 devices on my C-3 hub. 52 Zigbee devices, 7 Z-Wave devices, 54 Lutron devices, and a variety of Hubduino, Alexa TTS, Sonos, and Harmony Hub devices.
@Csteele Is 16 days your longest stretch of uptime? My hub ran 18 days on .145 then needed a reboot. Up until then I could have been convinced that 'the problem' had been solved. My expectations might be a bit out of sync with everyone else's.
Ever? no..
For v2.2.3.148, I think it was released on the 16th. I don't think it's possible to be on .148 and have a longer uptime.
I haven't yet upgraded to that version; will probably do so in the near future.
