I use my hubitat in my summer house and use it for controlling ZWave plugs that in turn control electric heaters, all controlled via Virtual Thermostats based on input from some temp probes.
At some point during the last 6 days, my hub must have locked up and it seems this may have happened at a time when the heaters were heating. This then resulted in the heaters continually heating the rooms up to 40°C (104°F). This is of course not ideal, both from the standpoint of potential damage, fire hazard and electricity costs.
I only found out about this today when the home mode did not turn on in preparation for me going to the summer house. I logged on to my network and I could not log on to the UI. I was able to logon to 8081. I then used CURL to reach /hub/reboot. Once I had done that, and only a few seconds later, i was able to logon to the UI and it updated my SharpTools dashboard.
Of two evils, I think it was better to have it stuck on heating, rather then having it stuck on off as then I would have had frost related issues (burst pipes etc.)
So, due to this, I have some questions:
How can I better monitor if the Hubitat is responding correctly from my internal network ? It did respond to pings, so that is not a way to check.
Can I / Should I do a regular restart from within the Hubitat ?
When I do a POST to /hub/reboot, what does it do ? Surely it does not powercycle the hub, because it was way to fast to respond again.
Can I tell anywhere when the hub started to not respond ? There is nothing in the logs. I did check on the backup/restore page, and it seems that no backup has been completed since the 9th of March, and then only just now, so perhaps this is a "good" way to gauge when it hung ? i.e. ~5 days ago ?
What could have caused the hung state ? Perhaps logging ? If so, is there a way to easily, centrally, turn off all logging for devices ?
I see some have setup a Wifi plug to be able to restart the Hubitat, but I have read that a sudden power loss may not be best for the hub. Due to this I have also bought a UPS, so that would rule that out anyway.
This is what I do for Hubitat monitoring in NodeRed. It's not perfect (as you said, it DID respond to a ping), but I find that querying the Maker API (one could also just check the main URL as well as they run in the same process) is usually a good indication of the hub status. Typically, if you can reach 8081, then using the /hub/reboot URL solves any lockups.
As I understand it, POSTing to /hub/reboot does a soft reboot of the Java services running on port 80. Support would be able to answer that definitively though.
I wouldn't advise against using a Wifi switch as hard booting the hub does generally cause more issues.
I don´t have any knowledge of Node Red, but I understand the diagram and I will have to do something similar I think. The problem I have been having though is that I have a small PC on my network for doing tasks like this and I have been unable to disable power save reliably, so it stops responding once in a while. To keep it up, i am using pings from Hubitat, so you see where this is going
I do have a RPi that I wanted to use for Hubitat logging, but it seems that introducing that into Hubitat (IncfluxDB / Grafan) will also cause issues.. everywhere there are small mountains to climb..
What I have done is taken a RPi and loaded NodeRed on it (most people that know me know that I'm a NodeRed junkie for home automation stuff). It's a rabbit hole, but once you get the hang of it, it's actually really easy and fun to use with thousands of different NPM plugins available.
For log shipping, I like to use what's native in Hubitat: Web sockets. There's a couple of undocumented websockets available for events and logging (ws://[hubitat_ip]/eventsocket and ws://[hubitat_ip]/logsocket respectively). I'm using the events socket to trigger things in NodeRed and I have a small nodejs app running that listens on the logging socket and just stuffs those into a MySQL database running on the RPi as well.
Once you have the logging data in MySQL, you can use anything you want to graph/search off of it.
.. I'm using the events socket to trigger things in NodeRed and I have a small nodejs app running that listens on the logging socket and just stuffs those into a MySQL database running on the RPi as well.
Just curious - couldn't you use NodeRed to push the logs into MySQL as well? Also what about testing the event socket for lockups. The http way does seem easier though.
That occurred to me yesterday as well and I'm actually working on that right now. It was one of those "DUH" moments. LOL
The problem with using event socket as a check is that it is output only from HE and not able to be triggered. The Maker API is a GET, so it can be triggered any time.
disclaimer: As the web sockets are undocumented in HE, there could be an server socket running that no one knows about and support is hiding from us... kinda like aliens as they are probably hiding them from us too. lol
In fact, in this case if the web server is still alive enough to respond on port 8081 and act on the reboot url, then it must have just been the java app that was locked up. So even a software watchdog should be able to detect this and restart it.
I am not all that good with node-red and after trying to re-create this for the last 3 hours I figured I would check to see if you could export and/or give some more details on how you have this setup. Otherwise I'm afraid my RPi might have a very up-close and personal meeting with a hammer here in a few minutes. thanks!
Yes, the electric heaters do have their own thermostats, but in my infinite wisdom I just cranked them all up since they are controlled via Hubitat. I have now set them all to a max temp so this will not happen again.
lol
thanks!
