Coding an HTTPS request with authentication

🚧 Developers
1

Hi community,

I am looking for what is probably the most basic of assistance so I apologise in advance. I am trying to code an application that integrates the myenergi solar equipment into the Hubitat ecosystem and I have fallen at the first hurdle, despite hours of looking around at examples across HE and ST. It is very frustrating because it seems it should be obvious...

What I am trying to do is to craft a simple HTTPS request to https://s7.myenergi.net/cgi-status-* and to pass the username and password (which I have already gathered from the user during app initiation). All of the apps/drivers that I have found seem to use an API key whereas this service requires the user/pass combination which is why I am strugging a bit to find an example :frowning:

I am more than happy to self teach so if somebody can just point me towards a simple example of an app that uses this type of authentication that would be very much appreciated.

TIA

1 Like
2

Once you pass the username and password in an initial API call, does it then provide you with a key to use in future API calls? I have used that kind of setup in a few of my drivers.

More generally, you may want to use an application like PostMan to test the API calls before introducing them into your code, I find it quite useful.

3

Thanks for the quick response. No, no API key unfortunately. I'll play about with PostMan to see if I can track down the specifics - I haven't done this before so embarking on a learning journey

4

Shame....

Here's a cut down example of some authentication I do in one of my drivers... There's a couple of references to variables and methods I haven't included, but hopefully it will make sense, if you can use it at all...

def retrieveAuthCode_KumoCloud() {

def vnewAuthCode = ""
def vbodyJson = "{ \"username\": \"${UserName}\", \"password\": \"${Password}\", \"AppVersion\": \"2.2.0\" }"
    
def postParams = [
    uri: "${getBaseURL()}/login",
    headers: getStandardHTTPHeaders_KumoCloud("yes"),
    contentType: "application/json; charset=UTF-8",
    body : vbodyJson
]
       
try {
    
    httpPost(postParams)
    { resp ->
        debugLog("${resp?.data}")
        vnewAuthCode = "${resp?.data[0].token}";
        debugLog("retrieveAuthCode_KumoCloud: New Auth Code - ${vnewAuthCode}");
        
    }
    
}
catch (Exception e) {
    errorLog("retrieveAuthCode_KumoCloud: Unable to query Mitsubishi Electric ${MELPlatform}: ${e}")
}
return vnewAuthCode
}


def getStandardHTTPHeaders_KumoCloud(excludeAuthCode) {

def headers = [:] 

headers.put("Accept-Encoding", "gzip, deflate, br")
headers.put("Connection", "keep-alive")
headers.put("Accept", "application/json, text/plain, */*") 
headers.put("DNT", "1")
headers.put("User-Agent", "")
headers.put("Content-Type", "application/json;charset=UTF-8")
headers.put("Origin", "https://app.kumocloud.com")
headers.put("Sec-Fetch-Site", "same-site")
headers.put("Sec-Fetch-Mode", "cors")
headers.put("Sec-Fetch-Dest", "empty")
headers.put("Referer", "https://app.kumocloud.com")
headers.put("Accept-Language", "en-US,en;q=0.9")

return headers
}
5

Thanks. Will dig into this later. I have a brain explosion happening right now - I have been staring at code all day... :slight_smile:

1 Like
6

If the username/password is passed with each request, it is probably using what is referred to as "Basic Auth", which concatenates username:password (note the colon in the middle), then base64 encodes the string. This is then passed in the request as a header: Authorization: Basic base64_encoded_string.

2 Likes
7

Thanks @dkilgore90 for your reponse. Testing the API using Postman shows me that it requires Digest authentication, the call fails using Basic. Is there a similar option to the above but using Digest? I have a suspicion the answer will be be no but here is to hoping!

8

No experience with "Digest Authentication" -- but the concepts are outlined here, hope this helps! Digest access authentication - Wikipedia

9

This might help:

There would be some work required to port to Hubitat and your specific use case. But the general structure might be a good reference.

10

This looks really promising. I'll have a play. Thanks.

I'll post the ported code if and when I manage to get it sorted.

@dkilgore90 FYI I had found https://stackoverflow.com/questions/45361721/groovy-digest-authentication as a pointer to digest auth but haven't managed to get much further as yet.

11

Nope, still struggling I'm afraid. Here is what I think that I know (feel free to shoot me down where I have it wrong):

  1. The HubAction class (method?) example from @tomw and the subsequent parsing of the response is only applicable to device drivers - I am trying to code an app example (at least initially)
  2. Digest Auth requires a two part request- the first request requires you to interrogate the response headers to obtain the nonce. It also returns a 401 which indicates that authentication is required. You can catch the 401 and interrogate the reponse header to get the value that you need.
  3. The second request then uses that nonce value and provides the credentials to the server which authenticates and returns the complete headers and data

I am stuck at interrogating the returned request from the first call. I simply dont know enough about groovy coding and am not able to find a similar example online to serve as an example. I have used the httpGet method and encapsulated this in a try catch construct (sorry if these are not the right terms). I do get a 401 response but it is handled in the exception (catch) part of the code and I don't know how to use that to interrogate the headers and obtain the nonce value.

2021-08-01 13:44:09.993 debug Exception: groovyx.net.http.HttpResponseException: status code: 401, reason phrase: Unauthorized

What I am trying to do is to connect to an API that requires digest auth and obtain a string of a server that can then be used in further API calls. Details are at Myenergi API. The end game is to have parsed the asn value from the returned header, which would contain the server string to be used in all other calls (also digest auth) but I am having no luck at all.

Reaching out to the developer community in the hope that this sparks somebody's imagination enough to give it a go and start me in the right direction. If I can get a working example of a digest auth call in an app code I would more than likely be away. I am more than happy to learn but I have really hit a brick wall here in my knowledge of HE dev and groovy in general.

All contributions gratefully received.

12

I trap HTTP 401 in a few of my drivers and re-auth (for an expired JWT in my case, but the same general structure should probably work). Here's an example -- see httpExecWithAuthCheck() in hubitat_unifiEvents/unifiController at main · tomwpublic/hubitat_unifiEvents · GitHub

For synchronous HTTP operations (like httpGet()), Hubitat returns an HttpResponseDecorator. You can get the headers from that response with <response>.getHeaders().

It's only slightly more complicated with async HTTP operations, as described here: Async HTTP calls

13

Looks like the HTTP headers are what you need to extract, based on your original post and the linked post (https://s7.myenergi.net/cgi-status-*), seeing that 401 is an "expected" response with key info.

https://director.myenergi.net/cgi-jstatus-*

Simon

1 Like
14

I left my post a little light... looks like you need to identify the "asn" in the headers and use that in subsequent HTTP calls I expect...

15

As I thought about it some more, I think you will have to use the async HTTP calls, because the 401 response will throw an exception immediately from the synchronous HTTP call, and you won't be able to get at the response and headers.

Here's a stab at how to handle this based on reading the Myenergi thread that you linked to:

def callApi(someJsonBody)
{
    def params =
        [
            uri: "https://director.myenergi.net",
            body: someJsonBody
        ]
    
    try
    {
        asynchttpGet(handler, params, [body: someJsonBody])
    }
    catch (Exception e)
    {
        log.debug "failed: ${e.message}"
    }
}

def handler(response, data)
{
    if(response?.getStatus() == 401)
    {
        log.debug "response headers = ${response?.getHeaders()}"
        def params =
        [
            uri: "https://" + response?.getHeaders()?.getAt('X_MYENERGI-asn'),
            body: data?.body
        ]
        
        httpGet(params)
        { resp ->
            if (resp)
            {
                log.debug "resp.data = ${resp.data}"
                doSomethingWith(resp)
            }
        }
    }
}
16

@tomw and @sburke781 thanks heaps for your help. I had to step away from the code for a bit due to time limitations but I had a good stab at it yesterday and have got the digest authentication working with async calls (good tip @tomw). If you are interested in how I have done it (and this may well be the only publicly available example of digest auth working in Hubitat at the moment - not saying it is a good one though!) then the code is at https://github.com/VeloWulf/myenergiIntegration. It is definitely still a work in progress but the basic structure around the digest auth is in place.

The next problem that I need to overcome is that by doing all of the calls as asynchronous it means that I struggle to interact with the UI. The specific example that I have is the loading of devices into the app config pages but I am certain that there wil be others down the road. I am going to try reworking the code to use synchrronous calls and rtap the 401 error (and hopefully the associated headers) in the exception, store the info in the state variable and then reuse that same info is the subsequent calls to both the director server (to obtain the ASN) and the main data server that has the hub data I need. Going to use the

technique described here by @tomw. Fingers crossed...

Do you have any tips for how to deal with the async calls and interact with the UI that I mightnot have thought about? The crux of it all is (obviously) that the async code continues to run after functions that call the code have completed. They would need to be told to wait for the async code to finish somehow but I haven't worked out a solution for this yet.

Thanks again for your help

2 Likes
17

Nice! To your async or delayed return question, I haven't had great success with the app config pages since you don't have as much control over when they load. I usually just brute force it and give the user an extra page to click through or an input that triggers a refresh of the same page to get updated data values into the input fields.

For an example of the latter:

dynamicPage(name: "mainPage2", title: "", install: true, uninstall: true)
    {
        section
        {
            input name: "beaconsToWatch", type: "enum", title: "Select all devices that indicate presence at your location.", options: state?.knownBeacons?.sort(), required: false, multiple: true
            input name: "refreshList", type: "bool", title: "Refresh this list", defaultValue: false, submitOnChange: true
            input name: "selectAll", type: "bool", title: "Select all devices?", defaultValue: false, submitOnChange: true        
        }
        
        if(refreshList)
        {
            app.updateSetting("refreshList", false)
        }
        if(selectAll)
        {
            app.updateSetting("beaconsToWatch", state.knownBeacons ?: [])
            app.updateSetting("selectAll", false)
        }
    }
2 Likes
18

Yeah, unfortunately I can't claim any experience or knowledge in app's myself, sorry.

19

I now have it working in a synchronous call and I have the second server (the referred asn) returning data. Woohoo!! The latest code is at the github link. It isn't too pretty, I'm afraid. In particular the pollASNServer function that returns the body data sends back something that looks like this, which is nested maps and lists all over the place. The code that I have written tried to parse it and it works (I think) but was done through a lot of trial and error and reading of online knowledge sources. If anybody has a better way to get to the data do let me know!! I am trying to identify which devices are active in the system (have nested values - just the eddi in the example below).

[
[eddi:[
		[sno:12345678, 
		dat:15-09-2021, 
		tim:05:48:32, 
		ectp2:-1549, 
		ectp3:3516, 
		ectt1:Internal Load, 
		ectt2:Grid, 
		ectt3:Generation, 
		bsm:0, 
		bst:0, 
		cmt:254, 
		dst:1, 
		div:0, 
		frq:50.02, 
		fwv:3200S3.048, 
		gen:3516, 
		grd:-1525, 
		pha:1, 
		pri:1, 
		sta:1, 
		tz:9, 
		vol:2326, 
		che:7.56, 
		hpri:1, 
		hno:1, 
		ht1:Tank 2, 
		ht2:None, 
		r1a:0, 
		r2a:0, 
		r1b:0, 
		r2b:0, 
		rbc:1, 
		tp1:62, 
		tp2:127]
	]
], 
[zappi:[
	]
], 
[harvi:[
	]
], 
[asn:s18.myenergi.net, 
fwv:3401S3.051]
]
20

Have you tried something like:

HashMap responseMap = (HashMap)evaluate(response)

which could theoretically allow you to reference the values semi-directly like:

myFwv = responseMap.eddi.fwv