The other possibility is that your reboot threshold might be rebooting the hub before Java GC reclaims the memory you think of as leaked. I don't know the parameters that they've set on the JVM for this. However, I'm not doubting the possibility of some state errantly being kept around after a backup (and thus not freed; i.e., leaked); I've worked on too many huge Java applications that I know this can accidentally happen.
I see memory drop (and not recover) after cloud backups. It is only like 5K though for me, so not much, I do not see that on local backups. YMMV.
I only do cloud backups weekly, so not an issue for me. Just chiming in with another data point.
I do something similar. The first backup on both systems is locally to the hub and the second is via a script being run on a Unraid server. The first one is around 2:15am. The second one is around 4:40. In the graph you can see the drop on the devhub(yellow) with each backup. But that is because the hub was just rebooted yesterday to load a beta firmware. Because of how little resources it uses java probably just chewed up memory for it. The Green line is my prod hub with a ton of stuff running my entire house. You can see it's backups at the same time and it dropped and then recovered immediately. because it has been up long enough now it is hovering between 130mb and 150mb of memory. It has stabilized in that range. Now would I like that to be higher certainly, but that is where it has stabilized.
We sometimes forget that memory is handled differently on different platforms too. Not using it can also be wasteful as well. The goal should be to load as much stuff in memory as possible, but then keep a small reserver so that you can handle small bursts of need. Then when that burst is done return to keeping as much data in memory as you can for your regular tasks.
2 Likes
I find a couple of things
-
when you update your hub and reboot, the update process runs stuff that uses more memory
- I find if I reboot again after an update (say 10 mins later), there is now more free memory than after the hub update and reboot
-
I find that if you are not using hub mesh, ensure it is turned off. Ie it being installed (and not being used) still uses a fair amount of memory. If you turn it off, reboot your hub.
I find these give me the most free memory. There may be other memory leaks/consumption. Part of this consumption should be expected as the JVM keeps trying to optimize and pre-compile code for apps you are using (hence turning off apps you are not using is important)
4 Likes
@nh.schottfam makes an interesting point about Hub Mesh. I don't use it on my Production array of Hubs... not for any good reason. I have been using HubConnect since before it was released and the whole process of stripping it out and then adding in Hub Mesh, which only gets me back to where I am today... isn't worth it to me.
However, although it's not clear that it would be a difference that matters, it is potentially a difference between your experience and mine. Grasping at straws though. 
1 Like
In case anyone missed it, there was a possible fix for a memory leak during cloud backups in 2.3.5.138, just released: Release 2.3.5 Available - #18 by gopher.ny
4 Likes
Here.is hoping it helps during regular backuos as well. Not using hub mesh and it is disabled.
If you see hub mesh under
HE console -> apps, then it is installed
go to the gear icon next to it and select remove (if you know it is otherwise turned off)
ie if you see an app under HE console -> apps, it is installed....
Not installed
Hub mesh does not show up in the Apps list.
You have to go to Settings, then Hub Mesh and can disable there.
1 Like
@gopher.ny Can you confirm if this fix is only for cloud backups or if it would impact memory after local backups as well?
I just made a local manual backup before updating from 2.3.5.135 (C7 Hub) and the Hub Information driver happened to get a refresh during the backup process. Here is the the events recorded from before and after the backup completed (the manual backup took approx. 2 minutes from 09:58-10:00).
In my test environment, the effect of the changes on memory use during local backup was within the "noise" brackets. So, maybe, but I'm not so certain as to put it in the release notes.
3 Likes
I used to have a Rule Machine that did a reboot at an odd hour, once a week.
I think that I'm going back to it.
I have a C-5 that I use for control purposes (both radios are turned off). It hasn't been rebooted since mid April (I haven't been updating it)> I just checked, and the free memory was around 170KB. I rebooted it and the free memory went up to 551KB.
This continual decrease in free memory is still a significant issue.
.
My C5:
Date/time,Free OS,5m CPU avg
2023-04-26 05:54:57,671836,0.74
2023-04-26 06:00:02,485152,0.47
2023-05-14 04:15:19,390664,0.07
My C7:
Date/time,Free OS,5m CPU avg
2023-04-26 05:50:15,654360,0.7
2023-04-26 05:55:20,471548,0.4
2023-05-14 04:16:24,381404,0.04
No Cloud, all local & both on 2.3.5.130
not sure what you are trying to show.. since i brought this issue up.. memory dropping during normal running is not the same thing as a distince memory drop immediately after a backup that is not reclaimed.. it is normal for the hub to have memory go down as apps are loaded unloaded etc especially after boot.
So you are telling me it's normal over time to run out of memory?
no but going down to around 200 has been normal for awhile
1 Like
ok, I'll leave it and see what happens, Thanks!
I feel like it is normal for systems like this manage their memory this way, where you see a slow decline of "free" over time. If it gets to a critical level, memory will be freed up and the system keeps running.
If however there is a true leak where memory gets into a state that the OS thinks it is in use still, it will not claim it back as free. Then over time that could lead to out of memory problems.
3 Likes
