Http get decoding

mark.cockcroft · June 4, 2021, 12:44pm

Im trying to build a driver to import .ical data, ive got the below but im inable to debug the response data, response status is 200 and working but nothing else dose
try {
httpGet(reqParams) { resp ->
if(resp.status == 200) {
log.debug "${resp.status}"
//log.debug "${resp.responseData}"
//log.debug "${resp.data}"

        }
        else { // not 200

sburke781 · June 4, 2021, 12:51pm

So you're saying it drops into the else statement? I.e. does not output log.debug(resp.status)?

thebearmay · June 4, 2021, 1:06pm

Might try something like:

void myIcalCall () {
    Map params = [
        uri    : "http://.......",
        path   : "/......",
        headers: ["Cookie": cookie]
    ]
    asynchttpGet("getICalHandler", params)
}	
	
HashMap getICalHandler (){
	HashMap iCalMap = [:]  
	try {
		if(resp.getStatus() == 200 || resp.getStatus() == 207) {
			String wrkStr = resp.data.toString()
			wrkStr.each() {
				List dSplit= it.split(":")
				if(dSplit.size()>1)
					iCalMap.put(dSplit[0].trim(),dSplit[1].trim())
				else
					iCalMap.put(dSplit[0].trim(),null)
		} else {
			iCalMap.put("error", "Response: ${resp.getStatus()}")
		}
	} catch (ignored) {
		iCalMap.put("error", "General Failure" )
	}
	return iCalMap
}

mark.cockcroft · June 4, 2021, 2:54pm

Trying to shutdown my pi using this but can't get it to work in RM

Yes, the UI sends a post request to "http://your-ip/settings with
name = syscmd and value = poweroff to shutdown
or
name = syscmd and value = poweroff to reboot to reboot your device

tomw · June 4, 2021, 3:32pm

Maybe httpGet is throwing an exception, like Connection Refused or something, which means no response regardless of status. What happens in your catch block?

mark.cockcroft · June 4, 2021, 3:38pm

it was runing on the .data but i wasn't patient enough looking at the logs

mark.cockcroft · June 4, 2021, 3:45pm

this is a snip of the data returned

this is what your suggestion outputted

thebearmay · June 4, 2021, 4:35pm

Hmmm looks like either: the toString() is causing an issue, or the data returned is different than what I'm seeing (for example, nowhere in that data do I see an "@")

If you put a line that outputs resp.data (i.e. log.debug "Data: ${resp.data}" ) what does that look like?

mark.cockcroft · June 4, 2021, 4:37pm

as the extract above, but it locks my hub for a good 5 min
respdataBEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:redacted@gmail.com
X-WR-TIMEZONE:Europe/London
BEGIN:VTIMEZONE
TZID:Etc/UTC
X-LIC-LOCATION:Etc/UTC
BEGIN:STANDARD
TZOFFSETFROM:+0000
TZOFFSETTO:+0000
TZNAME:GMT
DTSTART:19700101T000000
END:STANDARD
END:VTIMEZONE
BEGIN:VTIMEZONE
TZID:Europe/Lisbon
X-LIC-LOCATION:Europe/Lisbon
BEGIN:STANDARD
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
TZNAME:WET
DTSTART:19701025T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:+0000

mark.cockcroft · June 4, 2021, 4:41pm

this spec rfc2445
says
The iCalendar object is organized into individual lines of text,
called content lines. Content lines are delimited by a line break,
which is a CRLF sequence (US-ASCII decimal 13, followed by US-ASCII
decimal 10).

mark.cockcroft · June 4, 2021, 4:56pm

this py code has this - MagicMirror/calendar.js at master · MichMich/MagicMirror · GitHub

for (const e in calendar) {
const event = JSON.parse(JSON.stringify(calendar[e])); // clone object

do i need to JsonSlurper it?

thebearmay · June 4, 2021, 4:57pm

Okay, I see the problem - I was expecting one event and you're getting several. If we take each block from BEGIN:VEVENT through END:VEVENT and pass it through we could get a HashMap for the event...

Going to look for an iCal parser that could be used/converted...

mark.cockcroft · June 4, 2021, 5:12pm

ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(resp.data.getBytes())
locks the hub

mark.cockcroft · June 4, 2021, 5:23pm

there is this for majic mirro, but in in python

483 in calendar.js looks like its intrating over the list ....
for (const calendarUrl in this.calendarData) {
const calendar = this.calendarData[calendarUrl];
for (const e in calendar) {
const event = JSON.parse(JSON.stringify(calendar[e])); // clone object

			if (event.endDate < now) {

not sure if the problem is thats it is a ByteArrayInputStream, when ever i tyry to do anything the error keeps coming back to that?

def resptype= resp.data.class()
log.debug "resptype= ${resptype}"

thebearmay · June 4, 2021, 9:41pm

So a POC, where wkStr == resp.data.toString()

   wkStr = "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nCALSCALE:GREGORIAN\r\nBEGIN:VEVENT\r\nSUMMARY:Access-A-Ride Pickup\r\nDTSTART;TZID=America/New_York:20130802T103400\r\nDTEND;TZID=America/New_York:20130802T110400\r\nLOCATION:1000 Broadway Ave., Brooklyn\r\nDESCRIPTION: Access-A-Ride trip to 900 Jay St., Brooklyn\r\nSTATUS:CONFIRMED\r\nSEQUENCE:3\r\nBEGIN:VALARM\r\nTRIGGER:-PT10M\r\nDESCRIPTION:Pickup Reminder\r\nACTION:DISPLAY\r\nEND:VALARM\r\nEND:VEVENT\r\nBEGIN:VEVENT\r\nSUMMARY:Access-A-Ride Pickup\r\nDTSTART;TZID=America/New_York:20130802T200000\r\nDTEND;TZID=America/New_York:20130802T203000\r\nLOCATION:900 Jay St., Brooklyn\r\nDESCRIPTION: Access-A-Ride trip to 1000 Broadway Ave., Brooklyn\r\nSTATUS:CONFIRMED\r\nSEQUENCE:3\r\nBEGIN:VALARM\r\nTRIGGER:-PT10M\r\nDESCRIPTION:Pickup Reminder\r\nACTION:DISPLAY\r\nEND:VALARM\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n"


    log.debug wkStr
	HashMap iCalMap = [:] 
    Integer eCount = 0
    wkStr.eachLine{
        List dSplit= it.split(":")
        
        if(dSplit.size()>1){
            if (dSplit[0].trim()=="BEGIN" && dSplit[1].trim()=="VEVENT") eCount++
            iCalMap.put(eCount.toString()+dSplit[0].trim(),dSplit[1].trim())
        } else
            iCalMap.put(eCount.toString()+dSplit[0].trim(),dSplit[0].trim(),null)
    }
    log.debug iCalMap.toString()
    log.debug iCalMap.get("1DESCRIPTION")

mark.cockcroft · June 5, 2021, 7:30am

thanks for that, its kind of working, getting this error.

i think the problem is the description in the file, this is a extact from my calander, can we ignore/omit the fields taged as DESCRIPTION

CREATED:20210525T155038Z
DESCRIPTION:Good AfternoonMark\nI have been made aware that the joining ins
tructions for tonight's All Leaders meeting have not been sent out.\nI apol
ogise for this and hope it does not cause too much inconvenience.\n\nRather
than trying to rescue it at this late stage, I have decided to postpone i
t until next Tuesday 1st June at 7.30pm.\n\nPlease join using the following
link. Your section ADC's will be in touch if there are section meetings to
follow.\nThe GSLs meeting will follow on as normal.\nIf any GSLs were hopi
ng to raise questions tonight, I am available throughout the evening on th

if(resp.status == 200) {
log.debug "rest status${resp.status}"
wkStr = resp.data
log.debug wkStr
HashMap iCalMap = [:]
Integer eCount = 0
wkStr.eachLine{
List dSplit= it.split(":")
if(dSplit.size()>1){
if (dSplit[0].trim()=="BEGIN" && dSplit[1].trim()=="VEVENT") eCount++
iCalMap.put(eCount.toString()+dSplit[0].trim(),dSplit[1].trim())
}
else iCalMap.put(eCount.toString()+dSplit[0].trim(),dSplit[0].trim(),null)
}
log.debug iCalMap.toString()
log.debug iCalMap.get("1DESCRIPTION")

        }
            else { // not 200

sburke781 · June 5, 2021, 7:36am

Not sure if this is helpful or not, but I did have problems with one API call when trying to output it to the logs, it reformatted it, removing important special characters, so through off my assessment of how I needed to treat it in the code. Perhaps looking at the code through Postman or something similar may help...

mark.cockcroft · June 5, 2021, 9:08am

ive found a parser

github.com

peterbraden/ical.js/blob/master/ical.js

(function(name, definition) {

/****************
 *  A tolerant, minimal icalendar parser
 *  (http://tools.ietf.org/html/rfc5545)
 *
 *  <peterbraden@peterbraden.co.uk>
 * **************/

  if (typeof module !== 'undefined') {
    module.exports = definition();
  } else if (typeof define === 'function' && typeof define.amd === 'object'){
    define(definition);
  } else {
    this[name] = definition();
  }

}('ical', function(){

   // Unescape Text re RFC 4.3.11

This file has been truncated. show original

mark.cockcroft · June 5, 2021, 9:43am

moved on abit but their is just 2 much data (its only a 350kb file)

wkStr.eachLine{
                    if(!it.startsWith(" ")){
                    List dSplit= it.split(":")
                    if(dSplit.size()>1){
                        //log.debug "dsplit= $dSplit"
                        //if (dSplit[0].trim()=="BEGIN" && dSplit[1].trim()=="VEVENT") eCount++    //.startsWith(" ")
                        if (dSplit[0]=="BEGIN" && dSplit[1].trim()=="VEVENT") eCount++    //.startsWith(" ")
                            if (dSplit[0]=="SUMMARY"){
                                
                        iCalMap.put(eCount.toString()+dSplit[0].trim(),dSplit[1].trim())
                            }
                        //log.debug "ical map >1= $iCalMap}"
                    }
                    else {

thebearmay · June 5, 2021, 10:23am

Looks like the description may have contained a “:”

What you may want to do is narrow it down to the fields that you really want to see, and only add those to the map, i.e. Start TimeStamp, End Timestamp, Location, Summary. Probably should also refine the map structure similiar to

[events:[0:[start:yyyymmddThhmm, end: yyyymmddThhmm, location:xxxxxxxxxxxxxxxxx, summary:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx],1: [start:yyyymmddThhmm, end: yyyymmddThhmm, location:xxxxxxxxxxxxxxxxx, summary:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx],…]]

Means we need to interogate the dSplit[0] and little more, but ultimately you end up with a data structure that is more usable and could be more easily processed;

iCalMap.events.each{
…
}