Will the nodeJS based hubconnect server will be optional, and only used for websocket based connectoins?
Of course... The traditional http local, http cloud, and Websocket will still be available. This adds an additional option for those with a large number of devices and hubs as it acts as an event router and filter.
I hope to run this on my Synology NAS.
1 Like
I warm my pi up ready 
That's an excellent question...
As we talk up all that is new, there's a long list of all that is the same, that gets no mention. 
One of the many goals @srwhite has with HubConnect is to be as 'light' (efficient) on the hub as possible. The code is ultra optimized, and the introduction of HubConnect Server on an external NodeJS, takes that another step For Those That Need It.
I have 180 ZWave and Zigbee devices. It's not a small system, but compared to @srwhite and his 500 devices, mine isn't large. But mine is large enough to benefit from the use of HubConnect Server. The benefit is the offloading of the filtering that must occur.
This is precisely what I build the server component.. In my topology most of my devices flow inward to a central hub.. In otherwords, the server hub has a virtual copy of every device in each of my remote hubs, while my remote hubs only have a handful of devices shared back from the server hub.
The Hubitat websocket is very fast but it very, very inefficient... We learned recently that events are not deduplicated as they are in the event logs for each device... Second, event traffic on the server hub increases proportionately to each hub connected, thus increasing the number of events presented to each remote hub.
Consider this 3 hub scenario with hypothetical throughput.... (eps = events per second)
Server - 200 HubConnect Devices; 2 eps TX & 2 eps RX on the websocket.
Hub 1 - 100 Devices; 1 eps TX & 2 eps RX on the websocket.
Hub 2 - 100 Devices; 1 eps TX & 2 eps RX on the websocket.
Each remote hub is emitting 1 event-per-second. This means the server hub is receiving and parsing 2eps.. It also means it is emitting 2eps which each of the remote hubs has to receive and parse, regardless if they care about those events or not.
Add a 3rd hub in this scenario...
Server - 200 HubConnect Devices; 3 eps TX & 3 eps RX on the websocket.
Hub 1 - 100 Devices; 1 eps TX & 3 eps RX on the websocket.
Hub 2 - 100 Devices; 1 eps TX & 3 eps RX on the websocket.
Hub 3 - 100 Devices; 1 eps TX & 3 eps RX on the websocket.
So what is effectively happening is that even though you may be adding hubs to spread the load, you inadvertently add some amount of load tp all of your hubs by having to handle (i.e. parse) a lot of traffic for which they do not need to see. Eventually it can start to saturate the hubs ability to process events quickly which manifests itself in slow device updates.
The server has some basic statistical reporting that is made available to the remote hub device.
This is an average of events-per-second one of my remote hubs from the roughly the past 48-55 hours:
- proxyLoadRxPerSecond : 3.97 (events received from the server hub by the HubConnect server)
- proxyLoadTxPerSecond : 0.00 (events forwarded to the remote hub by the HubConnect server)
Essentially, my remote hub would normally be processing 4 eps, but with the filtering & deduplication, the number of actual events forwarded to the remote hub doesn't even register.
Bug/Misfeature Report
I just learned thee hard way that Device Names and Device Labels cannot contain the forward-slash character (/) when connecting said device to a SmartThings remote client (via http). If it does contain this character, you'll get error asynchttpGet() request failed with error 400 errors every time an update is being sent. This probably also applies to ANY http-linked client, but I haven't verified it.
FWIW, I can attest that it doesn't matter for socket connections - the device I was having issues was on another Hubitat hub (Hubitat_HubConnect_socket_Client --> Hubitat_HubConnect_socket_Master --> SmartThings_HubConnect_http_Client). Device Name and Label were both "Guest Suite Temp/Humidity Sensor".
Either the name should be encoded before the transmission, or there should be a warning when selecting devices with the '/' in their name.
1 Like
I've run into this as well (see post 1052). I wasn't sure if it was the server/client relationship that mattered, but what you discovered about HTTP vs. websockets certainly makes sense! My remote client was ST, so necessarily HTTP.
Trying to understand the nature of the issue..
Are you saying that SmartThings does not like device labels with slashes in them? It would be helpful to know exactly where does the error occur? The only time labels are transported between hubs is either saving devices or syncing a device, and in both cases they're conveyed in the response body as JSON strings where slashes should be legal. So I'm not sure where the issue really lies.
EDIT: For events, the server & remotes convert the event map into a JSON string then URLEncode it. So it's supposed to be encoded as '/' to %2F..
But the ST cloud could be decoding the URL and rendering it in the path. I know it is possible in Apache Tomcat for example...
I'll set up a test and see what's happening.
The error occurs on the HubConnect Server Instance for the SmartThings Client (I isolated the two errors below with blank lines...)
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:25:18.682 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Bedside Keen Vent [temperature: 59.67 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:25:16.502 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Adam's Bedroom Keen Vent [temperature: 82.25 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:58.182 pm [trace](http://192.168.1.77/installedapp/configure/39)Received ping from ChezBurke SmartThings (ST0).
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:45.831 pm [error](http://192.168.1.77/installedapp/configure/39)asynchttpGet() request failed with error 400
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:45.178 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Guest Suite Temp/Humidity Sensor [temperature: 69.19 °F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:41.434 pm [info](http://192.168.1.77/installedapp/configure/39)Received device update request from client: [3211, type TempRH]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:40.046 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Guest Suite Keen Vent [temperature: 76.72 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:38.862 pm [info](http://192.168.1.77/installedapp/configure/39)Received command from client: ["Guest Suite Temp/Humidity Sensor": refresh]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:35.200 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Treadmill Keen Vent [temperature: 90.50 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:33.297 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoSensor: Bar: [motion, active ]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:33.187 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoTherm: Downstairs: [temperature, 68.9 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:32.962 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoSensor: Media Room: [temperature, 68.8 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:32.761 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoSensor: 1st Floor Foyer: [temperature, 69.2 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:19.458 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Bedside Keen Vent [temperature: 82.25 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:24:18.882 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Adam's Bedroom Keen Vent [temperature: 71.61 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:57.126 pm [trace](http://192.168.1.77/installedapp/configure/39)Received ping from ChezBurke SmartThings (ST0).
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:45.588 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Guest Suite Keen Vent [temperature: 72.07 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:44.605 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Treadmill Keen Vent [pressure: 21.92 inHg]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:43.564 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Treadmill Keen Vent [pascals: 74241.5 Pa]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:41.711 pm [info](http://192.168.1.77/installedapp/configure/39)Received command from client: ["Guest Suite Temp/Humidity Sensor": refresh]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:40.650 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Treadmill Keen Vent [temperature: 63.77 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:25.814 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Sam's Bedside Keen Vent [temperature: 80.82 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:20.591 pm [error](http://192.168.1.77/installedapp/configure/39)asynchttpGet() request failed with error 400
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:23:19.828 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Guest Suite Temp/Humidity Sensor [temperature: 68.82 °F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:58.391 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Adam's Bedroom Keen Vent [pressure: 169.00 inHg]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:57.639 pm [debug](http://192.168.1.77/installedapp/configure/39)Sending event to ChezBurke SmartThings (ST0): Adam's Bedroom Keen Vent [pascals: 572315.1 Pa]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:53.920 pm [trace](http://192.168.1.77/installedapp/configure/39)Received ping from ChezBurke SmartThings (ST0).
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:32.451 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoSensor: Guest Suite: [temperature, 66.2 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:32.092 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoSensor: Sam's Room: [temperature, 64.4 F]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:31.796 pm [info](http://192.168.1.77/installedapp/configure/39)Received event from ChezBurke SmartThings (ST0)/EcoTherm: Upstairs: [equipmentStatus, idle ]
[app:39](http://192.168.1.77/logs#app39)2020-01-07 02:22:31.658 pm [info](http://192.168.1.77/installedapp/configure/39)Received command from client: ["2nd Floor Front": refresh]I just saw that.. The event is reaching the ST device so it's processing the event correctly... Just throwing an error for some reason.
If the event gets there, it does not complete updating the data - I found this after several days trying to figure out why temperature updates on the physical device were making it to the Server, but not onwards to the ST client.
No errors appear in the HubConnect Remote Client logs on the ST side...
I am afraid right now it appears that my theory is correct.. The SmartThings cloud is performing a URLDecode on URL's before passing them to the mapping handlers.
Hubitat is properly encoding the URL... So the device name is "aaa/Test Button" and the '/' is being escaped to %2F as it is expected to be.
This blob is passed as a parameter for event data...
%7B%22name%22%3A%22pushed%22%2C%22value%22%3A%221%22%2C%22unit%22%3Anull%2C%22displayName%22%3A%22aaa%2FTest+Button%22%2C%22data%22%3Anull%7D
I'm going to have to figure out a clever workaround for this SmartThings platform issue...
EDIT: Actually I'll just drop the displayName field from the event data. The receiving end always uses the local device label so the data is redundant anyhow.
Will that work if both the device.label and device.name include the "/"?
I added those to the event data a long time ago because I wanted to save the hassle of looking up the child device and taking it's label. Events only convey displayName which is one or the other.
It's probably best to add the child device lookup and use that information instead of passing it a URL param since ST is decoding URL's before passing them to the mapping handler.
The only other way is to do some kind of BBCode-esque translation internally.
EDIT: Both options have pros & cons.. I'll figure out a best approach and work it into 1.7..
HubConnect 1.7 Beta Update
I have received a good response to the invitation to participate in the beta so thank you for that. I haven't had a chance to respond to all PM's yet, so I apologize for that.
For the SmartThings issue reported today, I have a workaround for this quirk of the ST platform that will be included in the 1.7 beta.
Anyone who is concerned about monitoring hub slowdowns I've added an application-level ping monitor that could potentially help to monitor. The HubConnect server will send a message to the connected hub and the hub immediately echoes it back to the server. The difference is the RTT (in ms) for the message to make the full trip on the servers bidirectional websocket.
- proxyJitterMS : 127
- proxyLatencyMS : 12
The jitter value is a mesasurement (in ms) of the variance in the Hubitat websocket ping messages. This is a protocol-level event where the hub sends these at 120 second intervals. The server will track the interval between these. If this value is swinging wildly the hub could be experiencing heavy load.
This will hopefully prove to be useful in tracking hub slowdowns.
What am I missing?
I installed the HubConnect server components on my hub.
Installed the client components on a new hub.
Added a Zigbee RGBW bulb on remote and HubConnect driver on server.
Selected the bulb in the synchronize tab.
Server shows connected but I don’t know how to access the remote bulb. Both servers are running the current version (2.1.7.127).
Dunno what changed but now it is showing up.
FWIW, I know you try to avoid adding overhead whenever possible. Considering this, I would be happy with just a warning when I add the device. Or perhaps even an automatic translation of the name if the target is a SmartThings remote client. Some versions of Homebridge do exactly this, as Apple has a long list of "illegal characters" for HomeKit device names.
Is anyone here using Grafana and Node-Red? I'm graphing attributes from 100% HubConnect virtual devices on the coordinator hub. It's been working very well so far until I reboot the coordinator hub - then Node-Red gets stuck. I'm wondering if there's a way to automatically restart the flows, or see how other folks are doing it. (maybe this is a question for the Node-Red thread)
