When my garage door opens it plays a message on my Zooz thingy. I ran a quick errand this morning and when I came home wifey said she got no announcement. Went to check to see what happened. Could not access Hub. Nothing was working even things that didn’t need network access. Hub light was green. Tried everything and no luck.
At this point the time was Approx 7:15. (The importance of this will follow).
Finally pulled the plug as had no other options.
Once hub was back up I was able to access it again. Saw a bunch of errors in the log about things not executing, which yeah was true.
Looked at Hub events and it showed the following that occured at 7:02.
“Hub memory is critically low, rebooting now”
So if in fact it had rebooted at 7:02 then by 7:15 should I have not had access by then?
Under the Hub Events tab in the logs do you see a systemStart event around that time? (¿Possible that the hub wanted to reboot and didn't have enough memory to execute the command?) You should see 2 - the first from if the hub rebooted from the low memory and then the second from when you pulled the power.
Many people in your situation create a low memory rule to notify them before it hits critical. (I generally recommend using the Hub Information V3 driver to regularly check the memory (along with about 60 other POIs) - [RELEASE] Hub Information Driver v3).
I guess what puzzles me more than anything is why that’s happening. Reading all the posts about what can cause it and I don’t seem to have any of those issues. Virtually no logging going on unless something throws an error which is very infrequent. Can’t find any chatty devices. Only 163 total devices, 59 apps most of which are my own. 11 mb database size. Free memory test over 1 gig.
Just doesn’t make sense.
I installed the Hub Diagnostics app, but doesn’t really say what is going wrong.
Many moons ago when I used to write C++ code for Windows there was issues with the code, not my programs, that caused memory leaks. Spent many an hour talking to Microsoft about that. But in this day and age that kind of stuff should be solved.
Here’s a thought. If the Hub knows memory is getting low and automatically does a reboot, or tries to, maybe that threshold needs to be changed where it still has enough memory to do the reboot.
The problem is that this alert is generalized for all hubs. Your hub may work just fine with the alert on. Mine goes into low free memory alert for a number of hours almost every afternoon. It is running at less than 15% CPU load at that time.
Your hub only needs to reboot when the CPU load is high when there is a low memory alert. Then the memory reclamation process is overloading the hub.
Generally if I don't see any chatty apps or devices I start to look at any integrations/apps that utilize HTTP to communicate with devices or other platforms.
@JimB I think the threshold could be a bit tweaked to allow the App to determine what platform your on. EG; you’re on a c5 so that means you have 1G of Ram, blah blah - free percentage would be 15% before reboot trigger. If your on a c8, you get 10% of free before trigger, if your on a c8Pro 5%… (I’m guessing at the RAM sizes, I don’t really know what they are!)
Thinking about this some more and I’m a little confused.
the consensus is to use a rule to notify or reboot on low memory. But it seems to me that that is a built in function already. So why the need for a duplicate.
However, if the built in function can cause a complete hub lockup then it needs to be fixed. Everyone says don’t pull power to the hub without first doing a shutdown. But I had no choice.
I don’t usually connect to my hub on a regular basis unless I’m doing something or having an issue. And I can go days without ever looking at it.
Stuff happens that can casue corruption. I always suggest anytime strange things happen like sudden drop in memory or you get lock ups to try a reboot with the option to rebuild the DB or a soft reset.
It is always a good place to start and relatively harmless.
If nothing looks out of the ordinary then look at lan based integrations. They can be heavy handed.
The reason a rule is a good idea is because everyone's environment is different. Some folks can't have there hub under 180mb without issues, while some can run for weeks under 100mb. Then you have the option to control it yourself.
The key is the hub consumes memory as it needs it and then reclaim memory as objects in memory expire for a variety of reasons. If you dont do anything that causes a demand greater then avaliable memory then it will just continue to run.
I did create an app, I haven’t been using RM, and ran into something I hadn’t seen before.
I had an input with the name: C8Reboot. This was a switch to do a manual reboot. When I tried to do a subscribe to that it never subscribed. I finally changed the name to cReboot and then it worked. Is a numeral not allowed?
After you pull the power and reboot. That is to clean any DB issues caused by pulling the power, or by what caused the bad state to begin with. Any time you pull the power without a proper shutdown you should follow the next power up with a reboot with the advanced option to rebuild databased on reboot enabled.
Not do sound like a broken record but everyone’s setup is different. Technically you don’t have to worry about it. Eventually the hub will reboot on it’s own if memory is low enough.
A hard lock is a different issue though and a hard pull is really the only option, but these should be few and far between.
I have been monitoring the free memory on a daily basis. And it is gradually going down. I can't find anything in the logs, device stats, app stats, etc. that seems out of place. Nothing chatty that I can find.
I went from 1113332 on 7/29 to 1004384 today. Not much yet but steady down.
One thing to point out is the C8 Pro has 2GB of ram and your numbers are between 1.13GB and 1.00GB. That is allot of memory for this type of system. The system is probably just not being as aggressive with memory cleanup because it doesn't need to. Part of the calculation for memory cleanup is memory pressure. Your system isn't experiencing any pressure so it is only performing cleanup based on time.
I agree, but I started this thread because it eventually drops far enough to cause a reboot. So either memory cleanup is not working or something going on that I can’t find.
The saw tooth visual you can see above is what you are looking for. That would indicate the GC is working and recovering memory. That doesn't prevent tasks from grabbing allot of memory. That can be hard to see if you don't collect frequently enough though and that itself can be system intensive in itself. If you use @thebearmay hubinfo driver you could put memory on a short interval of 20 or 30 for a period of time to see if you can capture anything yanking a bunch of memory. Then atleast you may be able to get a hunch as to what is grabbing it.
How long does it last before you need to reboot because of low memory. I would expect you to be in the 100 to 200mb range before the hub would make you reboot.
