Simple event graphing using Linux tools and gnuplot

For those of you who can do simple programming under Unix/Linux, you can get quick-and-dirty gnuplot graphs of your Hubitat device event data from the real-time exported JSON records sent out by the Maker API App. [EDIT: A more efficient way to get JSON records is by connecting to the undocumented, unsupported websock event socket. See my posting later in this thread.]

You don't need to install node-red, javascript, or any databases, and you don't need to poll your hub. All you need to do logging is a machine somewhere that is running a web server, that lets you run CGI scripts, and lets you write to a file.

The Maker API App lets you set a URL to POST device events to. That means that for every device you select under Select Devices, every event those devices produce will be sent in JSON format to that URL in real time. That URL can be anywhere in the world.

Create a CGI script on the web server at that URL. Make the CGI script simply append the posted JSON records to an output file on the machine. No polling is needed, assuming the devices you are logging generate their own events periodically.

Since the JSON event logs produced by Maker API are missing a time/date stamp, I had my CGI shell script add one at the beginning using the sed program:

#!/bin/sh -u
# My URL for this script is: http://192.168.1.1/~idallen/makerapi.cgi

echo "Content-Type: text/plain"
echo ""

# Change the output file below to be where you store all your event data.
# Use $(date +%s --date='now UTC') if you want UTC time stamps (see `gnuplot` below).
exec sed -e "s/\"name\"/\"date\":\"$(date +%s)\",&/" -e "s/\$/\n/" \
    >>/idallen/makerapi.json 2>&1

As soon as you create the above script, make it executable, and put the URL for it into the Maker API App configuration, you'll start seeing JSON event data being appended to the file in real time for the devices you selected in the Maker API App configuration.

If you don't see anything, make sure your device is configured to generate its own events fairly frequently, and check your web server logs and web error logs for script and file permission errors. Make sure your web server is configured to permit CGI scripts in the directory you are using.

If you already have a database or existing graph infrastructure (e.g. Grafana), you could make the CGI script add the event data to the database directly, instead of putting it in a file. Perl has database tie-in functions; you could also use your favourite CGI language to do the same. I won't cover that here.

Here is a sample line from my generated .json output file (note the leading date field added by my CGI script above):

{"content":{"date":"1653841789","name":"power","value":"426.974","displayName":"Aeotec Energy Meter Gen5","deviceId":"1","descriptionText":null,"unit":"W","type":null,"data":null}}

Make a note to roll and empty this log file occasionally, or else you'll fill up your file system with device events.

With the JSON data being logged into a file in real time, all you need to do is pull the values you want to graph out of the file.

You could use the command line tool jq to convert the JSON records to CSV for input to your favourite CSV graphing program:

$ jq -r '."content" | [.date, .name, .value] | @csv' \
    < /idallen/makerapi.json >data.csv

I wanted to combine three JSON power entries onto one line for input to gnuplot, so I wrote a small Perl program to read the JSON records and output one line with the three power readings on the one line:

#!/usr/bin/perl -w
# Read the makerapi.json file and consolidate multiple power
# reports onto one line based on leading epoch seconds timestamp.

use strict;
use warnings;
use JSON;

my @powers = qw( power power-Clamp-1 power-Clamp-2 );
my %triple = ();
my $prevdate = 0;

while(<>) {
    next unless /^[{]/;
    my $objr = decode_json($_);
    die unless $objr;
    my $cont = ${$objr}{'content'};
    next unless defined($cont);
    die unless ref($cont);
    my $dn = ${$cont}{'displayName'};
    # only look at desired device
    next unless $dn eq 'Aeotec Energy Meter Gen5';
    my $name = ${$cont}{'name'};
    # only look at desired power fields
    next if ! grep(/^$name$/,@powers);
    my $date = ${$cont}{'date'};
    # if more than a second apart, start over
    if ( ($date-$prevdate) > 1 ) {
        %triple = ();
    }
    my $value = ${$cont}{'value'};
    $triple{$name} = $value;
    my $size = keys(%triple);
    # When we have all three keys, we can output them on one line:
    if ( $size == 3 ) {
        print "$date $triple{'power'} $triple{'power-Clamp-1'} $triple{'power-Clamp-2'}\n";
        %triple = ();
    }
    $prevdate = $date;
}

The above Perl script produces one line with three power levels on it:

$ ./i.pl /idallen/makerapi.json | head -n 4
1653841789 426.974 381.851 45.123
1653841794 427.62 382.428 45.192
1653841799 428.802 383.64 45.162
1653841804 430.294 385.181 45.113

That output can be put into a file and fed into gnuplot:

$ ./i.pl /idallen/makerapi.json >mydata
$ gnuplot -persist -e "
    set xdata time ; set timefmt '%s' ; set format x \"%Y
%m/%d
%H:%M\"  ;
    set grid ;
    set title 'Power consumption' ;
    plot 'mydata' using 1:2 t 'power',
         'mydata' using 1:3 t 'clamp1', 
         'mydata' using 1:4 t 'clamp2' ; 
 "

The X-axis graph times will appear in UTC unless you apply a time correction for your local time zone. One way to fix this is to adjust UTC to EDT by subtracting four hours in the gnuplot script itself:

    [...]
    plot 'mydata' using (column(1)-4*60*60):2 t 'power',
         'mydata' using (column(1)-4*60*60):3 t 'clamp1', 
         'mydata' using (column(1)-4*60*60):4 t 'clamp2' ; 

A better way to fix the UTC problem for gnuplot is to write UTC times in the CGI script. In the CGI script, change the command substitution $(date +%s) to $(date +%s --date='now UTC') and then you don't need to do the subtraction in the gnuplot script, nor do you need to keep track of Daylight Savings Time. This time adjustment may or may not be good for other plotting programs.

With appropriate scripting or gnuplot plot limits, you can limit the data plots to date ranges:

$ start=$( date '+%s' --date="May 29 3pm UTC" )
$ end=$( date '+%s' --date="May 29 5pm UTC" )
$ gnuplot -persist -e "
[...]
    plot [$start:$end] 'mydata' using (column(1)-4*60*60):2 t 'power',
[...]

Suggestion for Maker API: add a time/date stamp to the JSON records.

3 Likes

A more efficient way to get JSON records is by connecting to any of the several undocumented, unsupported websock sockets. Here's a little Perl script that does exactly that. You run it like this:

$ ./websocket.pl ws://hubitat.local/eventsocket >>$LOGFILE &

Since Perl does output buffering when writing to a non-terminal, you may have to wait a while to see any output in the output file. You can send the script a SIGUSR1 to flush output, or you can run it to the terminal and watch the events roll by.

This script understands the following undocumented HE websock socket record formats:

- ws://hubitat.local/eventsocket
- ws://hubitat.local/logsocket
- ws://hubitat.local/zwaveLogsocket
- ws://hubitat.local/zigbeeLogsocket
- ...most anything else...

It prepends the current date (GMT seconds since the epoch) to all records read. For eventsocket records beginning with source:, it also wraps the JSON with a content:{} outer layer to match what Maker API produces.

#!/usr/bin/perl -w
#  $0 ws://path...
# Listen on a websocket URL, e.g. ws://hubitat.local/eventsocket
# Touch up and then dump the records read to STDOUT.
#  - Add the current time in UTC seconds since epoch (for gnuplot)
#  - Wrap the JSON record with ("content": ... } to match Maker API JSON
# Traps SIGHUP SIGINT SIGTERM and flushes buffers before exit.
# Traps SIGUSR1 and flushes buffers and does NOT exit.
#
# Modeled after: Simple WebSocket test client using blocking I/O
#  Greg Kennedy 2019
# Originally from https://metacpan.org/pod/AnyEvent::WebSocket::Client
# Removed all reading of stdin; this version is listen only.
# -Ian! D. Allen - idallen@idallen.ca - www.idallen.com

use v5.014;
use warnings;
use JSON;

# Core IO::Socket::INET for creating sockets to the Internet
# Protocol handler for WebSocket HTTP protocol
# Time conversion to create UTC seconds since epoch
#
use IO::Socket::INET;
use Protocol::WebSocket::Client;
use Time::Local qw( timegm_nocheck );

die "Usage: $0 ws://URL\n" unless @ARGV == 1;

my $url = $ARGV[0];

# Protocol::WebSocket takes a full URL, but IO::Socket::* uses only a host
#  and port.  This regex section retrieves host/port from URL.
my ($proto, $host, $port, $path);
if ($url =~ m/^(?:(?<proto>ws|wss):\/\/)?(?<host>[^\/:]+)(?::(?<port>\d+))?(?<path>\/.*)?$/)
{
    $host = $+{host};
    $path = $+{path};

    if (defined $+{proto} && defined $+{port}) {
        $proto = $+{proto};
        $port = $+{port};
    } elsif (defined $+{port}) {
        $port = $+{port};
        if ($port == 443) { $proto = 'wss' } else { $proto = 'ws' }
    } elsif (defined $+{proto}) {
        $proto = $+{proto};
        if ($proto eq 'wss') { $port = 443 } else { $port = 80 }
    } else {
        $proto = 'ws';
        $port = 80;
    }
} else {
    die "$0: Failed to parse Host/Port from URL: $url\n";
}

warn "$0: Opening blocking INET socket to $proto://$host:$port\n"
   if -t STDERR;

# Create a basic TCP socket connected to the remote server.
my $tcp_socket = IO::Socket::INET->new(
    PeerAddr => $host,
    PeerPort => "$proto($port)",
    Proto => 'tcp',
    Blocking => 1
) or die "$0: Failed to connect to socket for host '$host' port '$port':\n$@\n";

# Create a websocket protocol handler.
#  This doesn't actually "do" anything with the socket:
#  It just encodes / decode WebSocket messages.  We have to send them ourselves.
warn "$0: Creating Protocol::WebSocket::Client handler\n"
    if -t STDERR;
my $client = Protocol::WebSocket::Client->new(url => $url);

# Set up the various methods for the WS Protocol handler
#  On Write: take the buffer (WebSocket packet) and send it on the socket.
$client->on(
    write => sub {
        my $client = shift;
        my ($buf) = @_;
        syswrite $tcp_socket, $buf;
    }
);

# On Connect: this is what happens after the handshake succeeds, and we
#  are "connected" to the service.
$client->on(
    connect => sub {
        my $client = shift;
        warn "$0: Successfully connected to $url\n"
            if -t STDERR;
    }
);

# On Error, print to console.  This can happen if the handshake
#  fails for whatever reason.
$client->on(
    error => sub {
        my $client = shift;
        my ($buf) = @_;

        warn "$0: Error on websocket: $buf\n";
        $tcp_socket->close;
        exit(1);
    }
);

# On Read: Called whenever a complete WebSocket "frame" successfully parsed.
$client->on(
    read => sub {
        my $client = shift;
        local ($_) = @_;
        # Something is wrong with the charset of these records.  Fix them:
        if ( s/([^\xc2])\xb0/$1\xc2\xb0/g ) {
	    # warn "DEBUG Updated Latin-1 in $_\n"
	    #     if -t STDERR;
        }
        if ( s/\\u00B0/\xc2\xb0/g ) {
	    # warn "DEBUG Updated odd escape in $_\n"
	    #     if -t STDERR;
        }
        # Add the current epoch "time" to the Hubitat JSON record.
        # Convert the time to UTC for gnuplot.
	# We accept /eventsocket, /logsocket, /zwaveLogsocket, /zigbeeLogsocket
        my $time = timegm_nocheck(localtime());
        # Wrap an /eventsocket record with ("content": ... } to match Maker API
        if ( s/^\s*\{\s*"source"\s*:/{"content":{"date":"$time","source":/ ) {
	    # This is /eventsocket format, make same as Maker API JSON format
            print $_, "}\n";
            return;
        }
	# Get here for all other JSON formats
	if ( s/^\s*\{\s*/{"date":"$time",/ ) {
            print $_, "\n";
	    return;
	}
        warn "$0: Unable to parse: $_\n";
    }
);

# Signals caught to allow exit with buffers flushing
$SIG{'HUP'} = $SIG{'INT'} = $SIG{'TERM'} = sub {
    # Sends correct close header
    $client->disconnect;
    die "$0: Caught signal @_ $!\n";
};
# SIGUSR1 used to flush buffers without exiting
$SIG{'USR1'} = sub {
    $| = 1;              # flush the output
    $| = 0;
};

# Now that we've set all that up, call connect on $client.
#  This causes the Protocol object to create a handshake and write it
#  (using the on_write method we specified - which includes sysread $tcp_socket)
warn "$0: Connecting to client\n"
    if -t STDERR;
$client->connect;
# Now, we go into a loop, calling sysread and passing results to client->read.
#  The client Protocol object knows what to do with the data, and will
#  call our hooks (on_connect, on_read, on_read, on_read...) accordingly.
while ($tcp_socket->connected) {
    # await response
    my $recv_data;
    my $bytes_read = sysread $tcp_socket, $recv_data, 16384;

    if (!defined $bytes_read) {
        next if $! =~ /interrupted/i;   # because of SIGUSR1 handler
        warn "$0: sysread on tcp_socket failed: $!\n";
        last;
    }
    if ($bytes_read == 0) {
        warn "$0: Connection terminated (EOF).\n";
        last;
    }
    # unpack response - this triggers any handler if a complete packet is read.
    $client->read($recv_data);
}
# Sends correct close header
$client->disconnect;

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.