Memory drop after backup

[csteele]

@kahn-hubitat

Well one part that rings true is there are memory leak(s) after nightly backups

I have 4 hubs in a Production array and also 2 Development hubs powered on 24x7 naturally. (This is to distinguish from the 4 additional hubs that are powered down / retired. )

None of them have a memory leak I've detected. That's NOT to imply there's no memory leak, it's to indicate the other end of the spectrum.. that inherently the hub has very low or no memory leaks. Leaks therefore would be configuration dependent. Your mix of Devices, Apps and Drivers exposes a leak that my mix of Devices, Apps and Drivers does not. And my point there is to say, it's going to be hard to reproduce. Quick resolutions occur when bugs are reproducible, while non-reproducible bugs take much longer.. so long they probably seem like they aren't being looked at.

[kahn-hubitat]

@csteele

That's NOT to imply there's no memory leak, it's to indicate the other end of the spectrum.. that inherently the hub has very low or no memory leaks. Leaks

Are you doing nightly backups and monitoring memory before snd after the backup. Its pretty obvious and i am not the only one who has reported it and posted memory graphs.

Here is another one. Also driver dependency would not really explain the drop after a backup. That almost certainly points to something in the system.

[rlithgow1]

Is it recovering after?

[kahn-hubitat]

No it drops slightly every night exactly after the backup as shown above. And eventually gets below my 150k threshold facilitating needing a reboot. Probably most people are not seeing it as new releases are usually coming out before it gets to that point.

[bobbyD]

@vadim_c

without anyone actually trying to reproduce it

Reporting something that one believes to be a bug is easy and open to everyone. While we make every effort to address an issue, often, the deciding factor on which issue gets resolved first, is the number of users affected.

The more users confirm an issue, the higher the priority its resolution receives. We get many reports from different threads, and even different channels, and we don't always respond to every case. However, that doesn't mean we are not actively monitoring a specific issue, or haven't tried to replicate the problem in our environments.

[mavrrick58]

The problem with reporting a Memory leak is that it could very easily not be the case. At the same time I can point out how my memory consumption goes over time on my prod hub I can show that my Dev hub stays 400mb even over a long period of time. The difference is what i am asking it to do. The work load on them is very different and speaks to what @csteele said above. It really comes down to what we ask for it to do. All those extra rules/automations and non stand drivers add up to create contention. Most of the time i see the nightly backup drop the mem down by like 70mb and then it recovers afterword.

[kahn-hubitat]

@mavrrick58

work load on them is very different and speaks to what @csteele said above. It really comes down to what we ask for it to do. All those extra rules/

Again i don't agree .. that would not the explain the drop Immediately after a backup that has nothing to do with what you are asking the hub to do.

I was a software developer and that type of diagnosis of behavior invariably points to a leak on that process that just completed. Ie the backup process.

[aaiyar]

@kahn-hubitat

Again i don't agree .. that would not the explain the drop Immediately after a backup that has nothing to do with what you are asking the hub to do.

FWIW, I am not seeing the same magnitude of memory loss. My backups happen at 2AM, and I see a small, but distinct, loss during the first backup after each reboot, but not at subsequent backups.

The loss I see occurs with C-8 (light blue) and C-7 (dark blue) hubs. So I tend to agree with @csteele and @mavrrick58 that memory loss does not completely correlate with running a backup.

[csteele]

@kahn-hubitat

I was a software developer and ...

Then I'm sure you know that Java is at least half the probability... I've heard no one answer this with "go tell Oracle."

Again, I'm not attempting to convince anyone that it's not an issue.. I'm simply trying.. in a long winded way, to say "it's probably complicated."

[kahn-hubitat]

Ya i believe you but in my case correlates 100% with a backup. I wonder if i have some sort of corruption?. I will try a soft reset and restore but that has been done in the past. .it would be nice if support would look at my engineering logs. In the past they found a leak from reviewing it that had to do with handles. My backups run at 6am

Ya i have a feeling it is some issue beyond hubitats control embedded in the os but it would be nice to know.

[mavrrick58]

@csteele

"it's probably complicated."

I think this is exactly it. There are allot of factors. I think it gets more alarming when you run a really busy hub.

@kahn-hubitat

i believe you but in my case correlates 100% with a backup

There is no doubt that memory is consumed with the backup. I do see a dip, but probably 9 out of 10 times within a few min my memory is almost completely back to what it was before. I do occasionally see it drop though, but if it isn't everytime i don't think it is a leak.

@csteele

Java

This is a big point to be made two. Java is funny about how it holds on to objects in it's teird memory spaces and kicks them out of memory as well. It doesn't always kick stuff out that you think it should.

[salieri]

@kahn-hubitat

it would be nice if support would look at my engineering logs

@kahn-hubitat - I sent a PM to @bobbyD when I had a similar issue after going as far as I could with the community here for help. He responded and took a look at the engineering logs. This would be my recommendation as he helped resolve my issue.

[kahn-hubitat]

i was wondering whats makes my backups different than other peoples.. its not a simple backup.. it is kicked off from my nas via the url so i have a copy nightly off the hub..

ie

it has improved in recent relases so maybe its getting better on how much is lost.. or how often.. fingers crossed because i didnt have the issue when on the c7. time will tell.

[jlv]

@kahn-hubitat

No it drops slightly every night exactly after the backup as shown above. And eventually gets below my 150k threshold facilitating needing a reboot.

The other possibility is that your reboot threshold might be rebooting the hub before Java GC reclaims the memory you think of as leaked. I don't know the parameters that they've set on the JVM for this. However, I'm not doubting the possibility of some state errantly being kept around after a backup (and thus not freed; i.e., leaked); I've worked on too many huge Java applications that I know this can accidentally happen.

[JasonJoel]

I see memory drop (and not recover) after cloud backups. It is only like 5K though for me, so not much, I do not see that on local backups. YMMV.

I only do cloud backups weekly, so not an issue for me. Just chiming in with another data point.

[mavrrick58]

I do something similar. The first backup on both systems is locally to the hub and the second is via a script being run on a Unraid server. The first one is around 2:15am. The second one is around 4:40. In the graph you can see the drop on the devhub(yellow) with each backup. But that is because the hub was just rebooted yesterday to load a beta firmware. Because of how little resources it uses java probably just chewed up memory for it. The Green line is my prod hub with a ton of stuff running my entire house. You can see it's backups at the same time and it dropped and then recovered immediately. because it has been up long enough now it is hovering between 130mb and 150mb of memory. It has stabilized in that range. Now would I like that to be higher certainly, but that is where it has stabilized.

We sometimes forget that memory is handled differently on different platforms too. Not using it can also be wasteful as well. The goal should be to load as much stuff in memory as possible, but then keep a small reserver so that you can handle small bursts of need. Then when that burst is done return to keeping as much data in memory as you can for your regular tasks.

[nh.schottfam]

I find a couple of things

• when you update your hub and reboot, the update process runs stuff that uses more memory

  • I find if I reboot again after an update (say 10 mins later), there is now more free memory than after the hub update and reboot

• I find that if you are not using hub mesh, ensure it is turned off. Ie it being installed (and not being used) still uses a fair amount of memory. If you turn it off, reboot your hub.

I find these give me the most free memory. There may be other memory leaks/consumption. Part of this consumption should be expected as the JVM keeps trying to optimize and pre-compile code for apps you are using (hence turning off apps you are not using is important)

[csteele]

@nh.schottfam makes an interesting point about Hub Mesh. I don't use it on my Production array of Hubs... not for any good reason. I have been using HubConnect since before it was released and the whole process of stripping it out and then adding in Hub Mesh, which only gets me back to where I am today... isn't worth it to me.

However, although it's not clear that it would be a difference that matters, it is potentially a difference between your experience and mine. Grasping at straws though.

[bertabcd1234]

In case anyone missed it, there was a possible fix for a memory leak during cloud backups in 2.3.5.138, just released: Release 2.3.5 Available - #18 by gopher.ny

[kahn-hubitat]

Here.is hoping it helps during regular backuos as well. Not using hub mesh and it is disabled.