[2.3.5.135][C-8]Memory leak?

I've been seeing a slow memory leak with the C-8 across recent revisions. It seems that 2.3.135 has exacerbated the problem. After ~48 hours, I was already at ~175 Mb of free memory and device responses/automations began taking a hit (slow running). I've reboot the hub and am back to a healthy ~430 Mb of free memory. I do plan to keep tabs on it.

C-8

I'm seeing a similar problem on my C-7, but to a lesser degree. I presume that's more because of the radios being disabled.

C-7

@support_team if anyone would like to check the hub logs, I can send my ID over.

NOTE: The spike in both graphs from Friday is the reboot after updating to 135. I have the ZigBee device polling disabled. I checked the logs around the last major dip and there's nothing out of the ordinary.

2 Likes

Seems like the leak i have been reporting now with multiple versions with no concrete response from support. In my case it occurs after each backup.

2 Likes

I think you're on to something. I have cloud backup set to 3:30 am weekly. That ran this morning and reports finishing at 3:32 am this morning. The big drop on the C-8 is reported at 3:33 am.

The only oddity is that I do not have cloud backups enabled on the C-7. The memory drop on the C-8 is pretty consistent, just also with that big dip when the cloud backup ran.

This is reproducible by manually running a cloud backup. Both dips in the graph are from doing so.

1 Like

I'll hazard a guess here, the memory leak is related to the core system running things on the hub, Java maybe or whatever they're using. Otherwise the HE staff have been pretty decent at fixing things, and I'd expect if it was in their control they'd have done something about it.
Does that sound close?

In any event I gave up and built a rule to reboot when memory is below ~240K

1 Like

Same but mine is 150k. But i don't think you are correct as i remember earlier times when the hub would stay up for a month. If it was baked into the os that would have not been possible.

2 Likes

OK - this verifies what I have been seeing.

(insert pretty memory graph here)

When the memory gets too low on my C7, the Z-Wave radio will become totally fubar. The only way to "fix" it is to power down for 3 minutes and restart the device.

If I can reboot before the memory gets too low, I avoid any craziness.

1 Like

Who knows? Anyone on here is just guessing. Only the staff would really know the answer to this.

3 Likes

After today's cloud backup:

To recap since I rebooted on Sunday:


@support_team

2 Likes

That makes sense, however that makes me even more curious as to why it hasn't been fixed permanently, although from time to time it gets better, than a few updates later memory issues. My device mixture/rules have been consistent, free memory has not. I'm still on 2.3.4.158 (C7) for concern about Zigbee issues. My free mem check rule winds up rebooting the hub about every 15-22 days

I'm seeing very similar results. I would occasionally see this on my C7, then an update would seemingly clear it up.

I've since bought a C8 for my main hub, and offloaded a lot of automations and cloud stuff to my old C7.

The C8 is still dropping pretty quick. I'm at 6 days and down around 180mb.

I don't have cloud backups, but I was running backups at 2:15am daily. I just changed that to 3am daily, once every 3 days.

Typically I'll back up manually before an update or before/after making a lot of changes.

image

I also don't know what the database cleanup at night entails, but in the past I've ran it manually via the endpoint then rebooted. After that I'll have the most free memory ever.

1 Like

Still tanking pretty hard. The drop at 3 am correlates to a local backup. I moved my cloud backups to 10:25 am. There's a smaller dip at that time.

4 Likes

Here is mine in the last 12 hours. I have two backups that take place. The first one around 2:15am is the local backup on the hub. The one around 4:40am is when my Unraid server runs a script and downloads a backup. The second seems to have recovered the memory. The first appears to have lost most of what was consumed. I think it is also interesting that both my dev hub and my prod hub had decent size dips. The dev hub doesn't have hardly anything on it. Why it would have a big dip is beyond me.

6 Likes

My backup plan is having my server use the URL in a PowerShell script to trigger a daily backup that it then stores in a Google Drive folder. I had cloud backups set to weekly on just the C-8.

Today, I sat down and moved every device off the C-8 that's not attached via the Z networks. Something like 75 devices. I also removed almost all apps. The only ones left are the Google Home community app (I'd rather shoot myself in the foot than set that all back up), my sensor grouping app, my energy cost calculator app, and the smarter humidity fan app. The rest are built-in (Rule Machine, Simple Rules, Notifications, Maker, LCM, and RL).

I also did a soft reset for good measure.

We'll see if anything changes.

3 Likes

This thread seems to have dropped off in May. Does that mean you are no longer seeing this problem or you have found some solution?

I am, and have been, have the same issue on my C7 for some time. It seems to be getting worse. I havenโ€™t graphed it but I think I will. What are you using to graph this? Hubigraph and Hub Information? Another method not on the HE device? The purple graphs look they are done on a phone or tablet.

LJ

There are lots and lots of posts about memory problems on 2.3.5.152. I have repeatedly looked for answers. Until the developers agree this is a problem and either find it themselves or give us the tools to find it, we're all kinda screwed. I've split the work between a C-8 and my old C-7 and I now can get a week out of the C-8, which is better than 3 days, but it's still nowhere near the month or two that I used to be able to go between memory related reboots. This is probably even more noticeable because firmware updates have been extremely rare lately. Bottom line: you are definitely not alone as a review of postings the past few months can attest.

One thing that was suggested to me was a soft reset to perform database cleanup. I was a little skepticle but tried it anways. It made a big improvement. I would suggest a soft reset

Soft reset seems a little drastic at this point. I have a rule to automatically reboot when Memory gets low and that is not too intrusive.

You say soft reset improves things. But it doesnโ€™t fix things, is that correct? How much did it improve things?

LJ

It improved things in the effect of how fast the memory drops. Things that would cause the memory to drop before are less dramatic. So in effect it helped the hub reduce the memory usage overall. A busy heavily loaded hub may not have as much iimprovement as a less busy hub.

This was initially done on my Dev hub which started to experience fairly rapid drops in memory. These appeared to be largely around Backups times.

My prod hub was done following the dev hub. It does seem a bit smoother, but is still dropping faster then the dev hub.

A soft reset isn't really a drastic move as it is basically just a backup and restore of the database. It doesn't impact your Zwave/Zigbee radios at all.

1 Like

My memory drop has improved (on the C-8 I mean..at 53 my own memory drop is more of an issue than firmware will resolve). I'm down to 195 Meg now so I'm hoping 2.3.6 beta isn't too far away as I'm stupidly trying not to reboot between updates (my reboot rule will kick in at 150 meg)

It's taken 45 days to get down to 195 Meg so I'm pretty happy with that.