EventStream / WebSocket closes immediately after connect to Hue bridge

🚧 Developers
1

I am hoping someone from Hubitat will pick this up and respond.

I have a Hue integration to solves some specific needs that the HE built-in integration did not support (not sure if it does now, have not tried it in a long while).

Hue has unofficially release push-notifications via the NChan HTTP multipart/mixed method for WebSockets.
Here is a write-up on the topic: https://githubmemory.com/repo/homegraph/philips-hue-push-client

I am in early experimentation to see if I can get HE subscribed to Hue push notifications.

This is what I used to subscribe:

interfaces.webSocket.connect("https://${host}/api//api/${state.username}/eventing/subscribe", pingInterval:  45, ignoreSSLIssues: true)

When it connects, I receive the following in webSocketStatus(message) call:

failure: Expected HTTP 101 response but was '200 OK'

As I understand it, the HTTP multipart/mixed method opens an http connection that remains open. And when events are thrown, the connection receives the data as a persistent stream. This is why it uses multipart/mixed, which is a MIME mail format for dividing an email into parts. In the hue solution, each event is a single mime part appended to the previous.

Technically, the asyncHttpGet would work, if we could obtain the reader from the connection, and parse messages from the reader as they arrive.

Anyone have any thoughts on how this might be possible with the current JAVA library restrictions HE puts in place?

2

Tagging @gopher.ny

3

I think @bertabcd1234 may have also been looking into this new Hue functionality as well?

4

I was, indeed. Hubitat has an EventSocket library, which is what I thought this connection was. I tried something like this:

interfaces.eventStream.connect(esURL, [headers: ["hue-application-key": hueUsername], ignoreSSLIssues: true])

and I get no errors, but the connection immediately disconnects. Staff were kind enough to add the ignoreSSLIssues flag here (analogous to ones for other HTTP methods), without which it predictably failed with a security error thrown before, but now I'm left with just this and nothing else in the logs to help.

image

If anyone figures this out, I'd be curious to know as well.

5

I missed the eventStream class in the documentation — I was too focused on the web socket. I will see what Incan find out about it. I agree that in the surface, this does seem like what it might need.

Where did you get the http headers info? I haven’t found the source yet to see how the hue push solution works (I haven’t looked hard yet). But it is interesting the I got an http/200 ok using the url I was passing. I am going to look into this route too. Thanks.

6

I think I stole it from the aiohue library that Home Assistant uses for their integration, which is the first place I had heard of this new API. It's possible another forum user also told me or hinted at this--I can't remember anymore. :smiley:

7

I tried using wireshark to capture the packets between the iOS hue app, and the hue hub, but there is no point, and I gave up, when I learned it only works over https.

I am digging further into this now. Thanks again.

8

It seems that the V2 API may have a different user token, and that is where we are failing. This is a total guess, but I do see that the V2 API is quite different, so it make sense we would have to re-authenticate with V2. Looks like I will be digging into the aiohue library next.

1 Like
9

Yeah, I can't figure it out. Everything is good. Either the API has changed, or there is an issue in the way the event stream works with the HUE stream.

either way, I guess i will just have to wait for the official release of the API once it gets documented.
I wonder if anyone else is using the EventStream mechanism.

My issue is the same, the stream is opened, and then closed immediately. No errors. I tried making a call to asynchttpget, but that just returns a time out immediately after opening the connection, even with a 60 second timeout.

10

It also matches what is used in the library that you originally linked to. If you can find a way to emulate this behavior using the functions available in HE: https://github.com/homegraph/philips-hue-push-client/blob/master/src/nchan-client.js

11

It looks like a persistent stream connection, so the Hubitat async http support probably isn't going to do it. Unless you can emulate the http layer with one of the socket interfaces, you may have to use something external to relay the messages to Hubitat.

12

I'm just making things up now and don't have a hue system to test on, so I'll probably make my exit after this, but...

Maybe try setting the Accept header as shown in the component that @armand linked to:

const options = { headers: { 'hue-application-key': this.apiKey, accept: 'multipart/mixed', }, https: { rejectUnauthorized: false, },

1 Like
13

Good guess! :smiley: I've tried that and a couple other things (including, at least, text/event-stream), and haven't been able to find anything that worked.

1 Like
14

What you are referring to is the text from the node.JS library. For reference, the full text is as follows:

  async connect() {
    const options = {
      headers: {
        'hue-application-key': this.apiKey,
        accept: 'multipart/mixed',
      },
      https: {
        rejectUnauthorized: false,
      },
    };

    return pipeline(
      got.stream(`${this.baseUrl}/eventstream/clip/v2`, options),
      this.streamSplit,
    );
  }

In the context of the options map, there are two options, headers and https. These are passed to the standard nodes.js got module to open a data stream to the target URL.
(see: https://www.npmjs.com/package/got)

got uses the first argument of the stream method as the URL, and the 2nd argument for function parameters (options). It is a common practice in Node.js to pass parameters as a map -- it is even part of the design pattern for React.js based projects as well.

So, they call got with two opens in the parameters argument: headers which are passed over the HTTP session to the server, and https which used internally by got as control flags for the https protocol. The https option rejectUnauthorized: false instructs got to trust all certificates.

Here is an excerpt from the documentation on got:

https.rejectUnauthorized

Type: boolean
Default: true

If set to false, all invalid SSL certificates will be ignored and no error will be thrown.
If set to true, it will throw an error whenever an invalid SSL certificate is detected.

The https:[ rejectUnauthorized: true ] is the same as HE's ignoreSSLIssues: true option

Given the above, that original source snippet essentially translates to:

asynchttpGet(huePushHandler,
             [uri: "${this.baseUrl}/eventstream/clip/v2",
              headers: ['Accept': 'multipart/mixed',
                        'hue-application-key': this.apiKey])

and this is essentially the same as what @bertabcd1234 tried to do with the EventStream class.

I used both accept content types (multipart/mixed, & text/event-stream). text/event-stream is the content type used in the aiohue library, and multipart/mixed is used by the homegraph/philips-hue-push-client. Both projects have the same solution, just different languages and different content types.

What these project do is to open an http session that is never closed. That session receives what amounts to a very large (endless) document that is sent slowly, over time. What our tests are doing is opening the connection, and then something is closing that session before we receive any data.
If it was working we should see that anytime we turn on or off a light, a mime message part should be sent with the details of the changes that just occurred, and the http session should remain open.

At this point all we can do is continue hacking away, or wait for the official release of the API for developer us.

15

Thanks again for the leads. The link helped a bit. Unfortunately, hue only uses https for this, so I cannot sniff too much from the hub. I am not interested rooting my hub to see what is inside, but that is essentially how this was reverse engineered from hue.

16

I have been digging through the Hue Developers guide, and it seems the API is already partially documented. It appears push notifications are part of the Hue Entertainment API. The feature is not documented in the API documentation, but there are hints around push notifications.

The API Key for the Entertainment API uses a client key (*see note), not the username. However, after much testing, I have confirmed this interface (event stream) does use username, not client key.

To obtain the client key, you must re-link the hub, and acquire the clientkey value from the response. To request a clientkey update the linking data packet as follows:

private requestHubAccess(mac) {

    String deviceType = '{"devicetype": "AdvanceHueBridgeLink#Hubitat", "generateclientkey": true}'

    asynchttpPost(requestHubAccessHandler,
                  [uri: "http://${this.networkAddress}/api",
                   contentType: 'application/json',
                   requestContentType: 'application/json',
                   body: deviceType])
}

def requestHubAccessHandler(response, args) {
    def status = response.getStatus();
    if (status < 200 || status >= 300) { return }

    def data = response.json
    if (data) {
        if (data.error?.description) {
            if (logEnable) { log.error "${data.error.description[0]}" }
        } else if (data.success) {
            def success = data.success
            if (success.username && success.clientkey) {
                state.username = "${success.username[0]}"
                state.clientkey = "${success.clientkey[0]}"
                if (logEnable) { log.debug "Obtained credentials: ${state.username} [${state.clientkey}]" }
            } else {
                log.error "Problem with hub linking.  Received response: ${data}"
            }
        } else {
            if (logEnable) { log.error "${data}" }
        }
    }
}

This key will be required for the Entertainment Streaming API of the Hue Sync, but it does not appear to be used by the Hue Hub directly for very much except Sync related actions. If you do not plan to implement the Entertainment API, there is no need to update your code to retrieve the client key. My code is update, as I would rather have it available if needed.

In my hue code, I added logic to test if the clientkey or username are missing, and if either are, to ask for a linking to occur again, and update the state. This does not affect the ability to use the hub in the old state, it is just available as an option if the user chooses to relink the hub.

[UPDATE:]
clientkey is NOT used for push notifications, I assumed it wass, because of the wording in the API documentation, but testing has revealed that it is not. I have not found a push notification document yet.

1 Like
17

Using curl on my computer, I learned it is the username, not the clientkey. Also, using the following curl command, the streaming works.

curl -k -H "hue-application-key: $key" -H "Accept: multipart/mixed" -X GET https://192.168.0.186/eventstream/clip/v2
--.tmB3dQ3b07cCuw/OAOSWDUQsxLomHim
Last-Modified: Thu, 19 Aug 2021 01:05:31 GMT
Etag: 0
Content-Type: application/json

[{"creationtime":"2021-08-19T01:05:31Z","data":[{"id":"9da4d2cf-5cdb-4678-8847-31fe196175af","id_v1":"/lights/13","status":"connectivity_issue","type":"zigbee_connectivity"}],"id":"690a8d87-8f90-4874-adaf-384fb6b796e5","type":"update"}]
--.tmB3dQ3b07cCuw/OAOSWDUQsxLomHim
Last-Modified: Thu, 19 Aug 2021 01:07:00 GMT
Etag: 0
Content-Type: application/json

[{"creationtime":"2021-08-19T01:07:00Z","data":[{"id":"78eb2160-08bb-4772-8107-c0bd4d5cc3df","id_v1":"/lights/16","on":{"on":false},"type":"light"}],"id":"48f08aa0-a23d-45e1-babe-9e5131384f6b","type":"update"}]
--.tmB3dQ3b07cCuw/OAOSWDUQsxLomHim
Last-Modified: Thu, 19 Aug 2021 01:07:01 GMT
Etag: 0
Content-Type: application/json

[{"creationtime":"2021-08-19T01:07:00Z","data":[{"id":"a2366ead-57f8-4e40-a476-7524f5807189","id_v1":"/lights/10","on":{"on":false},"type":"light"}],"id":"aaad3fb6-2fae-4147-9e98-3b80727f8c43","type":"update"},{"creationtime":"2021-08-19T01:07:00Z","data":[{"id":"e3edcc0a-cfaf-4c55-a6bd-13977a96fc49","id_v1":"/groups/13","on":{"on":false},"type":"grouped_light"}],"id":"01ca0a0e-84cf-4770-a01b-7c0add4d67f8","type":"update"},{"creationtime":"2021-08-19T01:07:00Z","data":[{"id":"3de03b87-0f72-4878-b0a5-9fe66401999a","id_v1":"/groups/14","on":{"on":false},"type":"grouped_light"}],"id":"d39ae478-ab29-489d-9b7e-c34be1abf52f","type":"update"}]
--.tmB3dQ3b07cCuw/OAOSWDUQsxLomHim
Last-Modified: Thu, 19 Aug 2021 01:07:04 GMT
Etag: 0
Content-Type: application/json

[{"creationtime":"2021-08-19T01:07:04Z","data":[{"id":"a2366ead-57f8-4e40-a476-7524f5807189","id_v1":"/lights/10","on":{"on":true},"type":"light"}],"id":"c431f48e-1065-430f-85ab-9a8c1411104d","type":"update"}]

This is only a small snippet of the output. The messages are deliver in real-time as the state of devices change.

Note: I put my hue username in a environment variable key so that I don't have to keep typing it, and to obscure it from this post.

This username is substantially different than any previous usernames the hub had generated. As such, I tried again with my new username, but no good. In the HE logs I just see:

image

I believe there is a problem with eventStream in HE, maybe it closes the connection because the stream is silent until an event occurs?

@support_team Can anyone provide some insight on why the eventStream closes the http connection but the curl command on my desktop, using the same headers, and arguments works?

Thanks in advance.

1 Like
18

Some additional details (verbose output from curl)

Using the exact same curl command as my previous comment on the subject:

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.0.186...
* TCP_NODELAY set
* Connected to 192.168.0.186 (192.168.0.186) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=NL; O=Philips Hue; CN=001788fffeacc32f
*  start date: Jan  1 00:00:00 2017 GMT
*  expire date: Jan  1 00:00:00 2038 GMT
*  issuer: C=NL; O=Philips Hue; CN=001788fffeacc32f
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc07c809200)
> GET /eventstream/clip/v2 HTTP/2
> Host: 192.168.0.186
> User-Agent: curl/7.64.1
> hue-application-key: MMilz*****************************iu70-JNe
> Accept: multipart/mixed
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Thu, 19 Aug 2021 01:28:58 GMT
< content-type: multipart/mixed; boundary=j4xU49jameUjHCC1Dk90nXZizeoB3Gio
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< content-security-policy: default-src 'self'
< cache-control: no-store
< pragma: no-cache
< referrer-policy: no-referrer
<

The streaming message body that follows looks like:

--j4xU49jameUjHCC1Dk90nXZizeoB3Gio
Last-Modified: Thu, 19 Aug 2021 01:29:19 GMT
Etag: 0
Content-Type: application/json

[{"creationtime":"2021-08-19T01:29:19Z","data":[{"id":"9da4d2cf-5cdb-4678-8847-31fe196175af","id_v1":"/lights/13","status":"connected","type":"zigbee_connectivity"}],"id":"b36a3991-187c-4f55-90db-b90462f7ba6f","type":"update"}]
1 Like
19

This stands out as particularly interesting. I wonder if this is related to the disconnect -- that it is using HTTP2 and re-issues the GET command via HTTP/2 protocol.

20

I'll check it out.

1 Like