C8 - Unexplained Steep Memory Drop 2.3.6.144

I've just noticed my free memory is a lot lower than usual and the hub will shortly need a reboot. Uptime is only 8.5 days. Looking at the free memory, I can see that all was well until the 12th at about 16:25 when there was a sudden unexplained steep drop from 272,000 to 183000. It hasn't recovered from that drop and has continued with its previous steady drop. There is nothing unusual in the logs for that point in time.

Any ideas what could cause this? I've previously gone over a month without the need to reboot and can't think of anything that's changed. Thanks.

Was a back up possibly taken at that time?

No - backups are at 2:15AM and I only do a manual one before an update. There was nothing going on, nothing excessively logging, no errors or warnings.

Looking at the entire Free Memory history, that 90,000KB drop that occurred in 5 mins that wasn't recovered is equivalent to what it used over a full 4 day period

Maybe post your logs showing a while before after that steep drop in case there is something there that might be interesting.

I've gone right back through the entire day and there's nothing untoward - weird

1 Like

It could just be that you're a bad person. The hub is very sensitive to personalities. :wink:

Was your sudden memory drop a one-time event as far as you know?

There is a benefit w/Hub OS memory to rebooting a second time after the initial reboot after a FW update. The second reboot will end w/you starting w/more memory than the first, due to how the hub processes apps/drivers on boot up.

2 Likes

Yes so far as I know it's a one off. The last couple of updates before this one I did the extra reboot. However later on, @gopher.ny said that wasn't necessary unless you needed that memory back. About 10 mins after a reboot my C-8 is at 400,000 and generally falls steadily and slowly, so I didn't bother last time. I just happened to notice that it was lower than normal and was surprised to see that large step of 90,000 for no particular reason.

Yeah, I remember Gopher saying that...problem is that you won't know you need the extra memory until you need it. :slight_smile:

I was thinking about looking to see if there is a way to programmatically reboot the hub automatically 10m after a FW update reboot...with a "hey - I'm about to reboot" text warning so I can cancel if I need to.

@danabw - I think you're on to something. The huge memory drop corresponds exactly to when I posted the below about a rival hub - the timestamp is in the image:

@bobbyD Did I hurt my C-8s feelings :joy:

5 Likes

LOL!! Smoking gun, dude!! :wink:

4 Likes

That should be possible, I might do the same. Perhaps if the version in Hub Info Driver was written to a variable on change (it looks as though Hub Info Driver updates the firmware version string as part of the 'Base Data' when the driver initialises at reboot):

Then that variable change can trigger a rule with a 10 min wait then reboot command.

1 Like

Scheduling reboot is not too difficult.. but..

There isn't an obvious way to handle this part... perhaps it could be possible with javascript or something injected into a dashboard, but could be tricky. (And require remembering to go to the dashboard to see and cancel it.)

This is probably a bit messy. I wasn't sure if I could do it all with local variables (are they maintained after reboot?) and also I don't know if the 10 second wait periods are needed/advisable.

3 mins after reboot, the fw version from hub info driver is written to the local variable (currentVersion) which is then compared to the hub variable (firmwareVersion). It exits if they're the same. However if they're different, currentVersion is copied to firmwareVersion and the hub reboots after a further 7 minutes.

I've added b******t values to the two new variables so that I can test it by rebooting. If it works I might add a notification and announcement with a few extra lines in the rule allowing me to abort the rule (I'll probably just do this by toggling a specific light on/off)

1 Like

I've tested that and it works as hoped. With variables purposely set incorrectly I manually rebooted. At start up it triggered and carried out a further reboot 10 mins later. After that reboot the rule correctly exited as there was no change to firmware version.

1 Like

I have my C8 set to auto reboot overnight when memory drops below 200000 KB and then reboot a second time 30 minutes later. After updates, since that is manual I don't want to follow with an auto reboot.

1 Like

@bbrannon I also have a rule that watches free memory. If it drops below 120000 (which I'm going to raise) and stays below for 2 hours, it notifies me on the phone and dashboard, waits until 3 AM and reboots the hub. The rule I posted above would only run at the first boot following a firmware update; the idea being to recover the memory drop that is not recovered automatically when an update takes place which Victor explained in another post.

This thread went a little off topic TBH as I started looking at how to automate that additional reboot. I never did get to the bottom of why I had such a steep drop in free memory that the hub didn't recover. It was after two days of run time with no errors, warnings or significant activity of any app/device.

What's concerning is that even with a rule to auto reboot when memory drops below 150,000 kb, such a steep instant drop as I saw would likely crash the hub, if that drop occurred when free memory was already quite low from it's expected steady drop.

1 Like

That happened around 4:20 in the afternoon. By chance could you have updated a app via HPM or through the manual process.

Or simply just recompiled a app.

More on track. I have not experienced any sudden steep memory drops but the overnight hub backup drop from which there is (usually) no recovery seems like it should be solvable by the devs. I t would be great if there were a way to determine and control what is actually being accumulated. I used to be good for a couple of weeks and now it is getting closer to a week and a half.

Nothing that I'm aware of. I updated Hub Info Driver via HPM but I'm sure I did that later, after rebooting the hub. I'll just keep an eye on it using:

http://192.168.0.50/hub/advanced/freeOSMemoryHistory

Having a peruse through that will indicate any repeats of the same that aren't obvious when just viewing the current available free memory value.

Not so long back I got to 38 days before it dropped below 180,000 KB