For some reason hubitat is crashing every night and I have to unplug and reset it every morning. Looking through the location events there are 3 webCoRE pistons that seem to be having “recovery events” every few minutes and in hub events it looks like the hub restarted almost 30 minutes before my good morning piston was set to run.
Looking at the piston logs is mostly unrevealing to me. However, there are a bunch of events in one of the pistons like this:
6/25/2018, 1:59:04 AM +956ms
+0ms â•”Received event [Lowanna Way].time/recovery = 1529892650913 with a delay of 13694042ms
+244153ms â•‘RunTime Analysis CS > 11571ms > PS > 10853ms > PE > 221730ms > CE
+274664ms â•‘Runtime (47513 bytes) successfully initialized in 10853ms (v0.3.104.20180323) (274662ms)
+274666ms â•‘â•”Execution stage started
+274905ms â•‘â•šExecution stage complete. (240ms)
+274940ms â•‘Setting up scheduled job for Sun, Jun 24 2018 @ 10:10:50 PM EDT (in 1s), with 1 more job pending
+295216ms ╚Event processed successfully (295215ms)’
It seems to be scheduling the job for a time that is already passed. And also, it doesn’t seem to be evaluating the piston, so I’m not sure what this log is really telling me it is doing.
I’m not sure where to start with this one. Thoughts?
If it's possible, pause the piston that is scheduling in the past and see if this resolves the issue.
FWIW I see piston recovery events in my location events as well but it dont think this should cause an issue. Its just webCoRE doing what it should do.
The slowdown is coming before his piston code starts executing:
CS - Child Start - Event received in child and the event delay is logged (now() - event.date)
PS - Parent Start - Child calls parent to get runtime information (commands, colors, variables)
PE - Parent Exit - Runtime information gathered, return to child
CE - Child Exit - Child variables initialized. Right before piston execution
RunTime Analysis CS < 11571ms < PS < 10853ms < PE < 221730ms < CE
Normally the exchange above is supposed to happen in ideally around 150ms, but it varies in hubitat between 300ms and 600ms if the hub isn't too loaded down. When the hub is booting up, I see delays on mine, but not to the extent of 5 minutes. If his hub is crashing, it sounds like resources are low and is reflected in the logs for the piston execution time.
I've found that if I have a piston for refreshing zwave devices (~30 devices refreshed every 3 minutes (1 every 8 seconds or so)) the hub will run out of memory and become unresponsive in around a day before needing to reset.
Has it occurred to any of you that WebCore is way overkill? Recovering missed scheduled events is so ST! You have a dedicated scheduler, it doesn't miss events. Bringing your hub to its knees with WebCore does not make much sense.
I had everything on webCoRE because it is so much easier to use than RM. (IMHO). One piston when I need 4 rules in RM in some cases.
That being said, I moved 4 of my simple pistons (some are triggered by motion) from webCoRE to RM and I must admit the motion rules do trigger quicker than they did on webCoRE.
I will transfer more over and probably leave my more complex ones on webCoRE.
Those are the three problem pistons, it seems. Also, I’ve noticed that it goes nuts after I switch my location mode to night mode. Prior to that everything is pretty calm.
Also also, all three of these pistons have a number of disabled statements that do not show up in these snapshots. That shouldn’t make a difference, but I’ll throw it out there if anyone thinks it is pertinent.
I’m using webCoRE 0.3.104, from ajayjohn’s GitHub repo. It’s running locally on my Linux server.
I paused those pistons and it still happened again last night. Now it looks like different pistons are in the list as having frequent recovery events.
In hub events there also appears to be a “system start” event every morning at 4:18 am.
I logged into the web interface last night right after activating night mode ad the whole thing became markedly less responsive. It seems to me that it is something about night and also guest mode that is problematic.
This could only be due to apps that you are running. Modes are simply a string held in the database, set by apps (or the web interface manually). There is no other functionality associated with modes other than what your own installed apps are doing, Care to show a list of what you have installed? Here or by PM.
So it should be obvious what app is responsible for your slow downs.
The unfortunate thing about WebCore is that it essentially created a programming environment on top of a programming environment (Groovy). This is always super inefficient. You are doing things with pistons that had they been done directly in Groovy would execute instantly.
All I can suggest is that you use other automation tools instead of WebCore.
I have around 42 pistons running without any issue other than the refresh but I had the same problem with refresh on rule machine and memory. I don't think it's webcore on it's own, but maybe a rogue piston causing trouble. I would look around any of the ones connected to mode changes. There's also some good tips for optimizing pistons on the webcore forums. I had to edit some to handle the resource constraints of the hub : moving multiple motion pistons to 1 that deal with motion, removing redundant checks for lighting, using changes to instead of if's for subscriptions, mostly checking the logs for slowdowns and editing appropriately.
Some users will also move their simple pistons over to rule machine and keep the complex ones in webcore. I prefer the UI of webcore and having a central place to keep all the automations, but with the platform you have a lot of choice to do what works best for you.
63 Pistons here. Everything is automated! Runs great. I love to tinker though so just today I moved about 20 lighting automations to RM. Just to see if there was any speed difference. So far it's working great.
RM is a powerful tool but the interface, in my opinion, is very dated and confusing. I was a long time Homeseer user (10+ years) and it just reminds me of HS 2 back about 7 years ago. HS changed to a setup similar to webCoRE a few years ago though (HS 3). So picking up how webcore works was very easy. Nice straight forward If Then Else type of setup. Very easy to follow. I don't mind learning a new system but RM just seems like a step backwards interface wise. Really don't want to hurt anyone's feelings on this because the power/usefulness is there...just needs a better interface.
Hey, won't hurt my feelings! I wrote it a few years ago for ST, when there was nothing like it. That was a very brief era. I pulled the app from ST after they totally broke the platform. CoRE was the result, filling the vacuum left. Obviously, Ady went way above and beyond anything that had been done before that in apps for ST.
No apologies for the interface, it's simply an artifact of what I was capable of at the time.
I used to have over 100 rules, but then moved things over to Motion Lighting, Simple Lighting, Mode Manager and Button Controller. My thought is the simpler the app that does the automation, the better.
Thanks for that. Sometimes words written in a forum can be taken the wrong way pretty easily. Definitely not my intention.
I joined the ST family just as you pulled the RM app and I just about dropped the platform because of it. There was no way to automate anything! Then core and eventually webCoRE came along and blew everyone away, lol. But even with that I always thought it was getting too bloated.
But back to the topic! @jrau272, the biggest thing I noticed on your piston screenshot...you have one labeled "Fuel Streams". Even on ST Fuel Streams was never finished. This could be a huge resource drain on your system. Definitely something to look at.
thanks for the encouragement everyone. I definitely prefer the webcore UI and like having all of my automations in one place.
@bptworld - yeah, fuel streams are definitely broken. That piston is actually paused, I was just playing around.
@jp0550 - I hope you're right that I just need to streamline some pistons. I've been trying to use fewer if statements. I'm also thinking that "stays inactive" might not be the best way to time having the lights turn off after a period without motion. I also updated to your most recent build and things seems a bit snappier, for whatever that is worth.
the location events seems to have quieted down quite a bit since I paused a few pistons. what exactly is a "recovery event" for a piston? those were happening like crazy before.
I seem to have paused or fixed enough pistons that the hub didn’t crash and was still responsive this morning. Only one webCoRE recovery event last night. And no system start hub event at 4:18.
However, I had a notification that the zigbee network was down - this was confirmed as none of my motion sensors were working. I’ve had that notification a few times over the past few days, but thought it was just because the hub was crashing, but perhaps that’s a different issue?
After a reboot (software, didn’t have to unplug at least), everything is back up and running again.
whelp. I found the problem piston. I had a while loop continuously evaluating fan speed. it was the bedroom fan that turned on when activated night mode. so it wasn't the mode change at all, those things just happened concurrently. I changed the piston to this:
which only evaluates when the fan initially turns on. I tried setting a timer to run the while loop every 15 min, but that caused everything to crash too.
I was running this piston on smartthings without much issue (at least it didn't crash or seem to slow things down much), but it seriously bogged down the hubitat. now everything seems to be running fairly smoothly.
Just kidding. Glad you figured this out.
