Logsocket stopped working after hub rebooted on its own

[rakeshg]

Hello - I recently created a flow in Node-RED that listens for log events using the websocket listener node and logs them to a database. It was working fine for the last 2 days. Today, at about 3:00 AM my hub restarted on its own (it's been a problem that I'm trying to track down).

Ever since the restart, there is no data flowing through to NR. I have modified the node and the flow but I can't see any data come through. HE logs are fine and have all the data. Any idea what could have happened? Or how to fix this?

All other Node-RED flows are working correctly. There is no change in the IP address (I have a fixed IP assigned). Any guidance would be appreciated - thanks!

UPDATE: Well - restarting Node-RED appears to have fixed it, but I'm not sure that is a long term solution. Does anyone know if I would need to restart NR every time the hub restarts? I don't know if this would also happen on a reboot

[martyn]

I've noticed this too, for some reason node-red thinks the socket is still connected but it isn't. If you look at the node-red websocket input node after a reboot of HE (or even a shutdown) chances are it still says "connected" but as you've noted no data is flowing.

Restarting node-red or anything that restarts the flow / node should bring it back - for example if you change the url in the node, deploy the flows, then change it back again and re-deploy it should be OK.

The node is supposed to reconnect if it detects that it has been closed ( Websocket node: add support for "reconnecting" to external WS · Issue #643 · node-red/node-red · GitHub ) but I suspect there's something wrong with the actual "is it closed" bit in the first instance. Or maybe it needs a "timeout reconnect if no data" callback adding...

[rakeshg]

@martyn

Restarting node-red or anything that restarts the flow

Yeah - that's what I suspected. Apparently there is an API to restart Node-RED flows, so I may try that. I already have flows that listen for the systemStart event from HE, so I could leverage those.

Restart flows using Node Red API

[toggledbits]

@rakeshg

UPDATE: Well - restarting Node-RED appears to have fixed it, but I'm not sure that is a long term solution. Does anyone know if I would need to restart NR every time the hub restarts? I don't know if this would also happen on a reboot

I had a slightly different solution for a project I'm working on.

As you said, I also noted that after a hub reboot, the eventsocket would connect but not pass data. I was tearing my hair out thinking I had screwed up the socket client on my end. I also had inconsistent results, which was maddening. I finally noticed that if I rebooted the hub and then immediately attempted to connect using a websocket test client plugin in Chrome, the socket was also "dead" -- connected but passing no data. This suggested my client implementation was not the problem, so I dug in further...

Anyway, to get to the point, I finally discovered that if you have "Hub Login Security" enabled, connections to the eventsocket will be dead until a hub login occurs. This distant dependency accounted for the "inconsistency" I had previously observed -- success or failure depended on whether I happened to log in to the hub or not, and when that login occurred relative to me reconnecting to the eventsocket. If the login occurred after connection, any connection made prior to the login never worked, but any connection after login worked fine every time.

My solutions:

1. Turn off Hub Login Security. This is clearly a reduction in hub security, but I don't see it as very strong anyway. Since most of my current users weren't having any issues, I suspect many people run this way by default.
2. Not use the eventsocket. That's undesirable, as the alternative for me was POST URL in Maker API, which I'm not a fan of (slower, less efficient, somewhat more fragile).
3. Incorportate a login into my HE interface. It turns out, requesting the login page (via HTTP), grabbing its session cookie from the response headers, and then POSTing a correct username and password (as form data) to the page (and passing the session cookie) gives the hub a login and subsequent eventsocket connections work as expected. This was easy for me to do as I'm coding in JS/nodejs, so I'm not sure how possible that might be to pull off in NR.

Edit: If any of the staff is reading this, it would be great if the eventsocket could just take a token like Maker API. I realize it's an unsupported/undocumented feature, but it seems that it's in such wide use these days, and a huge win over POST URL, that it should be promoted.

[rakeshg]

@toggledbits

Incorportate a login into my HE interface. It turns out, requesting the login page (via HTTP), grabbing its session cookie from the response headers, and then POSTing a correct username and password (as form data) to the page (and passing the session cookie) gives the hub a login and subsequent eventsocket connections work as expected. This was easy for me to do as I'm coding in JS/nodejs, so I'm not sure how possible that might be to pull off in NR.

Thanks for a great suggestion. I already have a process in Node-RED for logging into Hubitat with security (I use it to download backups, reboot the hub etc.), so I can create a flow that does an HE login after a systemStart event.

@toggledbits

I finally noticed that if I rebooted the hub and then immediately attempted to connect using a websocket test client plugin in Chrome, the socket was also "dead" -- connected but passing no data.

Hmm - that's what I am seeing. Node-RED showed it as connected, but nothing was happening. I am not sure how to "force" the Node-RED ws client to reconnect, so now I'm wondering if the login approach will work (since the connection was already established prior to the login). I guess I'm going to have to try it out.

[toggledbits]

@rakeshg

so now I'm wondering if the login approach will work (since the connection was already established prior to the login). I guess I'm going to have to try it out.

I'm still learning the ins and outs of Hubitat, but the hub login security, at least without SSL/TLS (is that even an option?), looks pretty weak to me, so not a big loss to turn it off and get it out of the way if you must, IMO. It's not even using the HTTP auth mechanism, so you don't even get the benefit of digest authentication or even silly base64 encoding of the password, it just goes in cleartext as form data, so any sniffing would reveal your login credentials very quickly. IMO if you're using hub login to keep your family from accidently reconfiguring the hub and playing with it, it's a serviceable feature, but as security for hardening your home IoT, not at all adequate. It's one of those "locks to keep honest people out" things.

[rakeshg]

@toggledbits

IMO if you're using hub login to keep your family from accidently reconfiguring the hub and playing with it, it's a serviceable feature, but as security for hardening your home IoT, not at all adequate

Completely agree with you and it's less about "security" than keeping someone from making unintended changes. Also, I have a bunch of stuff in Node-RED that uses the current login with security method (though all of them use the common sub-flow) that would need to be retested.

It looks like the login approach will work. I made no changes to my Node-RED process and rebooted the hub. There were nothing logged in Node-RED (using logsocket ws listener) after the reboot till some time after I logged into HE. I made sure I manually toggled some switches and nothing showed in the logsocket data. These events did show up in the MakerAPI client in Node-RED. But after the login to HE, data started flowing through!!!

Thanks so much for your help,

[rakeshg]

@toggledbits

Incorportate a login into my HE interface. It turns out, requesting the login page (via HTTP), grabbing its session cookie from the response headers, and then POSTing a correct username and password (as form data) to the page (and passing the session cookie) gives the hub a login and subsequent eventsocket connections work as expected.

Well - it turns out that logging in from a Node-RED flow does not work😟. There must be something different about logging in from the UI vs using Node-RED (I use the same login process to download a local backup and restart HE).

The other option that does seem to work is to "re-deploy" the Node-RED flow and I have a process that can do this unattended (I think). Going to try that next.

[ritchierich]

I have a rule that fires on hub restart to then restart NodeRed as I had a similar issue a while back. I also have a flow that will update a switch every 5 minutes as a checkin and if my hub doesn’t get an update from NodeRed it will alert me it’s offline. So have both sides covered.

To restart NodeRed I used a modified @ogiewon HTTP switch that I toggle. I had to modify his driver to pass extra query string parameters and a header required by NodeRed

Syntax to restart NodeRed is:
http://Ip-address:1880/flows with a header of:
node-red-deployment-type:reload

Happy to submit a pull request to @ogiewon if he’s interested in adding this functionality to his awesome driver.

[rakeshg]

@ritchierich

Syntax to restart NodeRed is:
http://Ip-address:1880/flows with a header of:
node-red-deployment-type:reload

This is great - the process that I was going to use (had it disabled till I tried the login approach) was passing the following:

msg.url = 'http://localhost:1880/flows';
msg.method = 'POST';
msg.headers = {};
msg.headers['content-type'] = "application/json; charset=utf-8";
msg.headers['Node-RED-Deployment-Type'] = "reload";
msg.headers['Node-RED-API-Version'] =  "v2";
msg.payload = {
    "flows": [
        {
            "type": "tab",
            "id": "7b994e31.f24a6",
            "label": "Hubitat Events DB"
        }
    ]
};
return msg;

I already subscribe to a systemStart event in NR and I was passing this into a http request node. Since this happens every time the hub restarts or reboots, I think I can do it in Node-RED. Thoughts?

[ritchierich]

@ritchierich

Happy to submit a pull request to @ogiewon if he’s interested in adding this functionality to his awesome driver.

@rakeshg I submitted a PR to @ogiewon's Hubitat HTTP Momentary Switch Driver adding the header option.

@rakeshg

Thoughts?

Personally I think its better to initiate the restart from HE as it is the one rebooting.

[ogiewon]

@ritchierich

I submitted a PR to @ogiewon's Hubitat HTTP Momentary Switch Driver adding the header option.

It has been merged. Thanks!

[ritchierich]

@ogiewon

It has been merged. Thanks!

No thank you for creating the original version!