Parse JSON response from GET

🚧 Developers
1

So, as many of you know I stink at web development and LAN code (I've self admitted that many times on here). That being said, I'll never get better if I don't practice.

So, one of the things I've been looking to do is get my Weatherflow weather station data into Hubitat "directly". I know I can do it all locally via WeeWX, or node.js, etc, but I thought I would try to do it as a web service.

Weatherflow has a REST API and I can get web service to respond with a JSON response on a crafted GET statement, so that's cool.

Being a JSON dummy, in general, I could use a pointer from there.

Any pointers on a simple driver or app that does a GET and parses a JSON response? I figure if I start with a working example, I can hack my way through this without asking TOO many dumb questions.

Edit: I'll start here. Should have searched more before posting.

An example of a value I'm looking to pull out is: "brightness":3

The JSON response looks something like this

{"station_id":xyzz,"station_name":"Name","public_name":"Name","latitude":29,"longitude":-98,"timezone":"America/Chicago","elevation":393.9548950195312,"is_public":true,"status":{"status_code":0,"status_message":"SUCCESS"},"station_units":{"units_temp":"f","units_wind":"mph","units_precip":"in","units_pressure":"inhg","units_distance":"mi","units_direction":"cardinal","units_other":"imperial"},"outdoor_keys":["timestamp","air_temperature","barometric_pressure","station_pressure","sea_level_pressure","relative_humidity","precip","precip_accum_last_1hr","precip_accum_local_day","precip_accum_local_yesterday_final","precip_minutes_local_day","precip_minutes_local_yesterday_final","wind_avg","wind_direction","wind_gust","wind_lull","solar_radiation","uv","brightness","lightning_strike_last_epoch","lightning_strike_last_distance","lightning_strike_count","lightning_strike_count_last_3hr","feels_like","heat_index","wind_chill","dew_point","wet_bulb_temperature","delta_t","air_density"],"obs":[{"timestamp":1561083194,"air_temperature":30.6,"barometric_pressure":961.7,"station_pressure":961.7,"sea_level_pressure":1008.1,"relative_humidity":81,"precip":0.0,"precip_accum_last_1hr":0.0,"precip_accum_local_day":1.490103,"precip_accum_local_yesterday":0.0,"precip_accum_local_yesterday_final":0.0,"precip_minutes_local_day":8,"precip_minutes_local_yesterday":0,"precip_minutes_local_yesterday_final":0,"precip_analysis_type_yesterday":1,"wind_avg":0.5,"wind_direction":84,"wind_gust":1.5,"wind_lull":0.0,"solar_radiation":0,"uv":0.0,"brightness":3,"lightning_strike_last_epoch":1560784918,"lightning_strike_last_distance":8,"lightning_strike_count":0,"lightning_strike_count_last_3hr":0,"feels_like":39.9,"heat_index":39.9,"wind_chill":30.6,"dew_point":27.0,"wet_bulb_temperature":27.8,"delta_t":2.8,"air_density":1.10294}]}

`

2

Take a look at the Ambient Weather app. It's doing the same thing of parsing JSON.

1 Like
3

Trying to visually determine the "location" of a particular value can be a real PITA. @ogiewon provided a great tool in the thread you posted in the OP that can map out the json structure for you. Unfortunately I'm in bed and the web tool isn't mobile friendly.

http://jsonviewer.stack.hu/

When you look at the examples of driver's etc you will notice that the returned json will be stored in "response.data".
You should then be able to get to the value you need by traversing the json structure one level at a time and the tool makes it a fairly straight forward affair.

How you plan to use the value you you retrieve is also important. I prefer to use async calls wherever possible. However, if you plan to have the results displayed in the UI (an input drop-down for example) the async is not optimal because you have to process the return data in a different function.

Only other advice I can think of is that logging is your friend. If you are not getting expected results, pump out a log for that "level" in the json and work your way back a level at a time till you do get an expected result. The elusive "[0]" has tripped me up a few times...at least before I got my hands on Dan's awesome tool above.
Good luck!

Stephan

1 Like
4

Today I'm all wx-ApiXU... :slight_smile:

from that source, first, start async, it's not hard to convert, but it's also simple to just start there.

def poll() {
	def requestParams = [ uri: "https://api.apixu.com/v1/forecast.json?key=$apixuKey&q=$zipCode&days=3" ]
	// log.debug "Poll ApiXU: $requestParams"
	asynchttpGet("pollHandler", requestParams)
}


def pollHandler(resp, data) {
	if(resp.getStatus() == 200 || resp.getStatus() == 207) {
		obs = parseJson(resp.data)
		doPoll(obs)		// parse the data returned by ApiXU
	} else {
		log.error "wx-ApiXU weather api did not return data: $resp"
	}
}

Set the URL into requestParams and fire off the query... eventually the website responds and as if by magic, the Handler runs. You check for valid and then put the json into a map. Other than the spelling test (get it all spelled the same everywhere.. resp vs response vs...) the Map has all the pieces.

obs.obs.brightness would have your value.

if (obs.obs.brightness) { etc...

or

if (obs.obs.brightness < 4) { etc...

As @stephack Stephack said: paste your Json into http://jsonviewer.stack.hu/ and then click "format" it's so much easier to see.

Two answers, both the same, just different slants. :slight_smile: You're rich!!

2 Likes
5

Thanks all. That really does help, and I appreciate you taking the time to write it!

Based on the examples, it does look pretty straight forward. I'm going to make a stab at it tonight or tomorrow.

Part of the reason for doing this, other than my own education, is that I don't have all of the parameters I want from my weather station in WeeWX anyway (lighting strike/distance, lux, etc). So I would need to dig into that to either extend the WeeWX DB, add a second DB, or map a few values into unused slots.

So I figured if I had to dig into something anyway, I might as well make it a more general learning session and try it via rest.

If that doesn't meet the performance I want, I'll likely step back and do it locally in node.js and shove it into virtual devices via MakerAPI. Or maybe I'll try to figure out how to get them in WeeWX. That looked a little annoying, though.

6

So as a follow-up... I decided to do this in node.js after all to keep everything local.

On my Unraid NAS/server I made a node.js docker container to run the node.js app. I'm fairly green on custom docker containers, but this one was really easy since it just needs to run the one node.js app file.

The node.js app is pretty simple. It just listens on UDP/50222 (weatherflow port) for the UDP broadcasts from Weatherflow and then uses MakerAPI to shove the values I need into a virtual omnisensor device.

Here is the node.js app in case it is helpful for anyone else. My code is sloppy, but it seems to work as I'm getting lux updates once per minute. Right now I'm only pulling in the illuminance/brightness value, I'll go back and add in the lightning strike data later as I have a few family things I need to run off to.

Weatherflow broadcasts the values once per minute, and I've found the lux readings both accurate and repeatable (and read up to 100K+ lux, which is more than anyone likely needs), so it may be useful for some lighting (or other?) automation - assuming the once per minute update rate is acceptable.

var http = require("http");

var PORT = 50222;
var HOST = '0.0.0.0';

var dgram = require('dgram');
var server = dgram.createSocket('udp4');

server.on('listening', function() {
var address = server.address();
console.log('UDP Server listening on ' + address.address + ':' + address.port);
});

server.on('message', function(message, remote) {
var obj = JSON.parse(message);
if (obj.type=="obs_sky") {
var lux = obj.obs[0][1];
var urldata = 'http://192.168.2.xxx/apps/api/xxx/devices/xxx/setIlluminance/'
urldata += lux;
urldata += '?access_token=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx';

  http.get(urldata, (res) => {
      const { statusCode } = res;
      const contentType = res.headers['content-type'];

      let error;
      if (statusCode !== 200) {
          error = new Error('Request Failed.\n' +
                  `Status Code: ${statusCode}`);
      } else if (!/^application\/json/.test(contentType)) {
         error = new Error('Invalid content-type.\n' +
         `Expected application/json but received ${contentType}`);
      }

      if (error) {
          console.error(error.message);
          // Consume response data to free up memory
          res.resume();
          return;
      }
  });

}
});

server.bind(PORT, HOST);

What the data looks like (is mostly cloudy today. :confused: ):

Capture
Capture1429×595 76.2 KB

1 Like
7

How do you like your WeatherFlow? I’ve always wanted a weather station and have done some research in the past but never pulled the trigger. New products have been introduced in recent years like this one and Bloomsky and others. One of my main desires is an accurate rain gauge.

8

Rain is the toughest measurement in some ways (windy/horizontal rain, etc).

The rain reading always shows rain when there is rain, which is good... How accurate it is, I don't know as I've never tried comparing it to another measurement.

I would say it seems "about right" though. :smile:

Weatherflow recently (last week or two) changed their entire rain calculation algorithm and biasing. Not sure if that will make it more or less accurate, though.

I pull the Weatherflow rain observation into my sprinkler controller (Rainmachine) which adjusts watering based on the value. Still have green grass, so it can't be TOO bad.

9

Very true. As long as it is close that is what I care about. I want to integrate it with my Rachio to better control schedule (via native Rachio integration and not HE). Thanks for the information!

10

Anyone have tips on how to parse this? Unfortunately I can't change the format.

    {"Status": [{"Engine": [{"Switch State": "Auto"}, {"Engine State": "Off - Ready"}, {"Battery Voltage": "12.90 V"}, {"RPM": "0 "}, {"Frequency": "0.00 Hz"}, {"Output Voltage": "0 V"}, {"Output Current": "0.00 A"}, {"Output Power (Single Phase)": "0.00 kW"}, {"Active Rotor Poles (Calculated)": "0 "}, {"Unsupported Sensors": [{"Raw RPM Sensor": "0"}, {"Frequency (Calculated)": "0.0 Hz"}, {"Unsupported Sensor 1": "0.00"}, {"Unsupported Sensor 2": "100"}, {"Unsupported Sensor 3": "0"}]}]}, {"Line": [{"Utility Voltage": "240 V"}, {"Utility Voltage Max": "242 V"}, {"Utility Voltage Min": "238 V"}, {"Utility Threshold Voltage": "156 V"}]}, {"Last Log Entries": {"Logs": {"Alarm Log": "07/01/19 17:10:45 Charger Missing AC : Alarm Code: 0000", "Service Log": "11/04/18 12:49:16 Schedule B Serviced ", "Run Log": "07/16/19 14:30:32 Stopped - Auto "}}}, {"Time": [{"Monitor Time": "Saturday July 20, 2019 17:47:33"}, {"Generator Time": "Saturday July 20, 2019 17:48"}]}]}

I loaded it up in http://jsonviewer.stack.hu/ and compared it to something I know how to parse (weewx json data), and can't seem to get very far.

I tried these for accessing 'Switch State':

       resp1.data.Status.each { engines -> 
            string Engine = engines.Engine
            Engines.each { states ->
                string SwitchState = states.'Switch State'
                string EngineState = states.'Engine State'
                string BatteryVoltage = states.'Battery Voltage'
            }
        }

and

        SwitchState = resp1.data.Status[0].Engine[0].'Switch State'

No luck thus far. Any tips would be appreciated.

11

Can you get the value for RPM? States.RPM?

If so then it is a parking issue with the space. If not I am not sure.

12

When referencing a variable in an object and you don't explicitly type out the key you need to put it within . The following should work:
witchState = resp1.data.Status[0].Engine[0]['Switch State']

13

No dice. I'm not sure .Status[0].Engine[0] is correct. It's almost like it should be resp1.data['Status'][0]['Engine'][3].RPM

Here is what resp1.data is:

{"Status": [{"Engine": [{"Switch State": "Auto"}, {"Engine State": "Off - Ready"}, {"Battery Voltage": "12.90 V"}, {"RPM": "0 "}, {"Frequency": "0.00 Hz"}, {"Output Voltage": "0 V"}, {"Output Current": "0.00 A"}, {"Output Power (Single Phase)": "0.00 kW"}, {"Active Rotor Poles (Calculated)": "0 "}, {"Unsupported Sensors": [{"Raw RPM Sensor": "0"}, {"Frequency (Calculated)": "0.0 Hz"}, {"Unsupported Sensor 1": "0.00"}, {"Unsupported Sensor 2": "65436"}, {"Unsupported Sensor 3": "0"}]}]}, {"Line": [{"Utility Voltage": "237 V"}, {"Utility Voltage Max": "245 V"}, {"Utility Voltage Min": "233 V"}, {"Utility Threshold Voltage": "156 V"}]}, {"Last Log Entries": {"Logs": {"Alarm Log": "07/01/19 17:10:45 Charger Missing AC : Alarm Code: 0000", "Service Log": "11/04/18 12:49:16 Schedule B Serviced ", "Run Log": "07/21/19 10:05:16 Stopped - Auto "}}}, {"Time": [{"Monitor Time": "Sunday July 21, 2019 13:29:20"}, {"Generator Time": "Sunday July 21, 2019 13:29"}]}]}

This is how it's different than my weewx json.

image
image826×866 88.8 KB

vs.

image
image802×1292 127 KB

resp1.data.stats.current.outTemp works great for that one.

14

Does this return anything? resp1.data.Status[0].Engine[0][0]

15

Nope. :frowning:

16

Have you tried drilling down one level at a time to see what is being returned? That's how I usually solve these strange json arrays. If the top level fails to return anything, then nothing below is going to work.
Start with resp1.data.Status to ensure you get something back...then work you way down the ladder.

17

Yep, I tried that. Never get anything to return, even if I do resp1.data.Status.

I did get it to work in Perl. This works:

$item->{'Status'}[0]->{'Engine'}[0]->{'Switch State'};

Which would map to something like this for Weewx Json:

$item->{'stats'}->{'current'}->{'outTemp'};

So from this it should be:

resp1.data.Status[0].Engine[0].'Switch State'

Also tried this with no luck:

resp1.data.Status[0].Engine[3].RPM
18

If this doesn't return anything, then everything deeper down definitely won't. Seems like your response may not be json. Post the entire method so we can see the http request...there may be a contentType mismatch.

19 
LOGDEBUG("Genmon: ForcePoll called")
def params1 = [
    uri: "http://${ipaddress}:${port}/cmd/status_json"
     ]

try {
    httpGet(params1) { resp1 ->
        resp1.headers.each {
        LOGINFO( "Response1: ${it.name} : ${it.value}")
    }
        if(logSet == true){  
       
        LOGINFO( "params1: ${params1}")
        LOGINFO( "response contentType: ${resp1.contentType}")
	    LOGINFO( "response data: ${resp1.data}")
        } 
        //   def SwitchState = resp1.data[0].Engine[0]
          //def SwitchState = resp1.data['Status'][0]['Engine'][3].RPM
        def SwitchState = resp1.data.Status
        LOGINFO( "Data: HERE2 ${SwitchState}")


[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.749 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Data: HERE2

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.747 pm [info](http://192.168.1.16/device/edit/1249) Gemon Driver - INFO: response data: {"Status": [{"Engine": [{"Switch State": "Auto"}, {"Engine State": "Off - Ready"}, {"Battery Voltage": "12.90 V"}, {"RPM": "0 "}, {"Frequency": "0.00 Hz"}, {"Output Voltage": "0 V"}, {"Output Current": "0.00 A"}, {"Output Power (Single Phase)": "0.00 kW"}, {"Active Rotor Poles (Calculated)": "0 "}, {"Unsupported Sensors": [{"Raw RPM Sensor": "0"}, {"Frequency (Calculated)": "0.0 Hz"}, {"Unsupported Sensor 1": "0.00"}, {"Unsupported Sensor 2": "65436"}, {"Unsupported Sensor 3": "0"}]}]}, {"Line": [{"Utility Voltage": "240 V"}, {"Utility Voltage Max": "242 V"}, {"Utility Voltage Min": "237 V"}, {"Utility Threshold Voltage": "156 V"}]}, {"Last Log Entries": {"Logs": {"Alarm Log": "07/01/19 17:10:45 Charger Missing AC : Alarm Code: 0000", "Service Log": "11/04/18 12:49:16 Schedule B Serviced ", "Run Log": "07/21/19 10:05:16 Stopped - Auto "}}}, {"Time": [{"Monitor Time": "Sunday July 21, 2019 19:57:17"}, {"Generator Time": "Sunday July 21, 2019 19:57"}]}]}

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.738 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: response contentType: text/html

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.731 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: params1: [uri:http://192.168.1.58:null/cmd/status_json]

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.728 pm [info](http://192.168.1.16/device/edit/1249)Genmon Driver - INFO: Response1: Date : Sun, 21 Jul 2019 23:57:17 GMT

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.725 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Server : Werkzeug/0.15.5 Python/2.7.16

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.722 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Expires : 0

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.718 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Pragma : no-cache

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.713 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Cache-Control : no-cache, no-store, must-revalidate, public, max-age=0

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.709 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Content-Length : 966

[dev:1249](http://192.168.1.16/logs#dev1249)2019-07-21 07:57:14.706 pm [info](http://192.168.1.16/device/edit/1249) Genmon Driver - INFO: Response1: Content-Type : text/html; charset=utf-8

Here is the raw output from Genmon: https://pastebin.com/raw/dQm6ABCL

20 
def params1 = [
    uri: "http://${ipaddress}:${port}/cmd/status_json",
requestContentType: 'application/json',
contentType: 'application/json'
]

Try that for your params and see if resp1.data.Status returns anything.

3 Likes