webCoRE for Hubitat Updates

#1

Overview

webCoRE provides a common programming environment and code sharing across different home automation platforms. webCoRE provides an integrated IDE, debug and trace environment for developing, testing, and monitoring your automations. It also has the ability to create dashboards.

  • This is the latest port of webCoRE onto the HE platform and has many optimizations for HE.

Current Release Updated December 9th, 2019

  • Improved logging
  • additional optimizations

Adrian presenting on webCoRE:

Previous Release Updated October 28th, 2019

  • Bug fix for cast errors
  • update to webcore Server code (for those running a local webCoRE server) for recent fix added

Previous Release Updated October 12th, 2019

  • Updated to work with latest webCoRE server updates to improve device loading, and support for more devices. This should alleviate most issues with having to open a small piston first.
    • Note if you use the public webCoRE servers, you need to be running this latest software
    • Note if you run a local webCoRE server, your local webCoRE server software MUST be updated to work with this new release
  • Optimizations, and fixes
    • Further performance improvements
    • HSM incidents / alerts reported via $incidents, $hsmTripped
    • added $weather support for Darksky weather data (must be configured in webcore application)
    • improved handling of webCoRE global and superGlobal variables
    • version numbers updated to meet webCoRE servers.

Installation

Note on Updating:

  • it is good practice to update the webcore-piston.groovy code first, then the webcore.groovy last when installing new files in HE.

  • Hygiene after an upgrade: going into the Hubitat Console and selecting: "Apps" -> "webCoRE" -> "Settings" -> "Cleanup and rebuild data cache" is a good thing to do.

  • HE has a limitation on the amount of data that can be returned on a http call from a server (128KB - headers). THIS SHOULD BE RESOLVED IN OCT 2019 RELEASE This can cause what appears to be a hang in the webCoRE dashboard (for example spinning when trying to load a piston) immediately after an upgrade.

    • I have found if you have a small piston, to open that piston first after an upgrade, as right after an upgrade the webcore server requests additional data from the groovy code. Once that data gets uploaded, it does not request it again (until next upgrade or login to the webCoRE servers).

    • If you have very large device lists, you may need to open this small piston twice (HE console -> Logs will show this) so that the IDE updates, and device list updates are all uploaded for this browser session.

  • Exception for Oct 10 update The local webCoRE software must be updated to work with this new release.

    • if you are running local webCoRE dashboards (this is an uncommon optional deployment), I would suggest updating webcore-piston.groovy and webcore.groovy and, ensure everything is ok with any code updates there, before updating your local dashboard.
      • this will allow you to more easily check things out in a new release.

Below is more details on the webCoRE port to HE

  • Here's a link to a thread that has links to the original github webCoRE code that was forked for use with Hubitat. It was this version of code that I started working with to optimize for Hubitat. Patched webCoRE for Hubitat (2018/09/09)

    • While a noble goal to keep the source code (groovy) in the hub the same as ST, this is impractical because of sheer code size, numerous platform differences architecturally, and the inherent higher horsepower ST has running in the cloud (Pistons don't compete with each other as much as they do on Hubitat).

      • separate hub code makes QA and release management easier, both on developers and users (users on one platform don't get changes destined for another platform).

      • we do want to keep the source / apis common on the webcore servers so that interoperability and piston sharing can work between platforms.

Features in this code :

  • This version is less overhead on HE platforms:

    • state (storage) reduction for webcore pistons

    • many parent/child calls have been removed

    • global variables as triggers only wakeup pistons subscribed to that variable

    • ifttt events only wakeup pistons subscribed to that event

    • reduced watchdog activity

    • reduce sending of location events (causes Hub db activity)

    • Should have less resource usage and execute statements much faster on Hubitat, especially if Parallel execution is turned (left) off for your pistons.

    • uses Async http calls for http commands

    • Pistons that run a long time will automatically "be nice" and release the cpu so other things can run / operate. Adds pause() for long running pistons to avoid CPU hangs of Hubitat. It will terminate long running pistons after 240 seconds.

    • Reduces thread usage by pistons significantly

      • Note Hubitat can run 4 threads at a time (cores) and has about 20 threads available for event processing.

  • Added new functions from latest webCoRE master

  • Has an HE version number in addition to webCoRE version

  • Ability run rules from Rules Machine.

  • Bug fixes

Note:

  • The piston bin (backup code) is located in state and settings variables for each piston (you can see these in Hubitat console by "Apps" -> select settings gear next to the application /piston.)

    • The backup code is also available in the webCoRE dashboard when you select the piston

    • it is good practice to enable automatic piston backups

webCoRE Differences / Restrictions (vs webCoRE on SmartThings platform):

  • Buttons operate differently in HE than ST, so when importing pistons from ST, these portions will need to be adjusted to the HE button devices and attributes.

  • HE port has removed code for lifx, and a lot of source comments (that caused problems compiling/saving on Hubitat)

  • Smart Home Monitor (SHM) does not exist in HE, but it does have Hubitat Safety Monitor. This is supported by webCoRE. Hubitat Safety Monitor API

  • Hubitat does not support $twcweather, so you would need to use a weather device for Hubitat with webCoRE or the experimental $weather support (below)

  • There is experimental support $weather based on apiXU (apiXU.com) data.

    • This requires installation of the webcore-storage.groovy app (HE Console -> Apps Code)
    • Enter your apiXU key in the webCoRE app (HE Console -> Apps -> webCoRE -> Settings)

  • sendPushNotification does work, it requires the HE mobile app, pushOver, twilio or some other notification driver on HE to be configured in the HE Console (Apps -> webCoRE -> Settings).

  • HE limits SMS messages to 10 per day, or you can subscribe to a separate service and use an appropriate device handler. HE HAS DECIDED THIS SERVICE IS GOING AWAY.

    • if your phone provider allows email -> SMS message, you can use webCoRE's email capability to send txt messages.

  • webCoRE on HE uses the 'us' (USA) region webcore servers. (I have not found a way to auto detect region on HE, and HE currently only has USA servers. (if anyone knows let me know via private message))

As mentioned in other notes, it is a good idea to control your use of:

  • enabling parallel execution in piston settings (under options when editing a piston), this should not be needed as Hubitat has much less available CPU than a cloud. (and this is a big hog of hub resources)

    • This causes webCoRE to use atomicState more which is slow in Hubitat

  • Control use of logging, both in code in your piston, and in the webCoRE dashboard when enabling logging on pistons. It is a good idea to leave it on "none" or minimal once you are done debugging a piston to reduce load on the hub and the database.

    • with the state size reductions in webCoRE, this setting is likely the largest consumer of state (db space)

  • Piston Design paradigms part 1: if a lot of things need to happen near the same time:

    • If you need a bunch of Pistons to run at the same time (say nightly or to run off the same trigger), you have a couple of design choices:

      • you can have a each piston with its own timer/trigger "At midnight..." (MORE OVERHEAD, as it causes a lot of threads to be consumed at the same time in Hubitat)

      • you can have one Piston called nightly work (or event work), and it calls each of the other Pistons that need to be run (BETTER in that less threads will be consumed (assuming you leave wait option set to true))

  • Piston Design paradigms part 2: In many pistons, you may be subscribing to more device attributes or states than you intend. If a piston is dealing with motion (and controlling a switch), it may not need to be subscribed to the switch (but commonly you may see it is).

    • Look for the "lighting bolt" in your piston code (in webcore dashboard/IDE) and ensure you need all the subscriptions that you see. These cause the piston to wake up (and consumes a Hubitat thread) for each wakeup.
    • Key is to choose your comparisons correctly (trigger vs. condition) - if the piston has a trigger, then conditions are not subscribed to.

  • Global variables - limit subscriptions (if you use multiple, see if you can subscribe to less variables as triggers)

    • This is a place where less is better - minimize total number of variables, and minimize changes and use as triggers.

  • I suggest to not enable log piston executions in the webCoRE app as this adds a bit of overhead on the hub and database (this is a webCoRE app setting; not a piston/code dashboard setting).

Installation Notes:

Reporting issues:

  • Apologies in advance if you have problems

  • A private message (here in this discourse forum) is many times best, and if you have a sample piston that shows the issue and/or logs, this makes things much easier to diagnose/fix.

If you need general webCoRE assistance on coding, examples, etc, the webCoRE community forum is a good place to visit, find programs you might reuse, have discussions with experts, etc.

Credits

  • Credits to all those who before me who wrote this, got it working, and keep it working

Links:

13 Likes
Using webCoRE with Hubitat
Community Apps
Homebridge Plug-in
Patched webCoRE for Hubitat (2018/09/09)
#2

Additional details on how things work:

  • Your pistons are stored and executed locally. the webCoRE servers provide IDE editing/debugging functions.

    • It is possible to run the webCoRE server locally (see below)

  • Currently webCoRE will terminate a piston if it runs for more than 240 seconds for a single event - so in addition to the "being nice", this should terminate a runaway piston.

  • When the webcore dashboard is open at the main dashboard page (listing your pistons), it updates every 10 seconds with Hubitat. While not a lot of overhead, be aware that it is updating

    • if you open the "Dashboard (beta)", there is much more overhead on Hubitat (while it is open)

  • Time Handling

    • HE platform handles time string parsing differently than ST. If you find time strings are not converting properly please report this (ideally private message, and logs and/or example piston).

    • When scheduling things that you want to run on "day changes" - for example midnight:

      • keep in mind that schedules can run up to a second or two early, so if you want to ensure it runs at or after midnight, set the piston to run "Every day at 1 minute after midnight", to ensure it runs on the new day.

    • the design paradigm of having a single "nightly" piston run, and it calls other pistons is more friendly to the resources on Hubitat.

    • If you are in an area that has daylight savings time (DST), Be careful scheduling work to happen during the change periods of DST, as things may not run, or run twice depending on the DST change happening. - 00:01 - 00:58 or 03:01 and later may be better choices.

  • If you need a larger application, you can use multiple pistons, and have a main piston call the sub-pistons. Again, leave the wait = true on when performing "execute piston"

    • do note:
      • to pass data to a piston, you can use arguments on the piston call, or global variables.
      • to get data returned from a piston execution, global variables are often the best choice.
        • global variables are "loaded" when a piston starts, so if piston A calls piston B, and B modifies a global variable, piston A will not see the change until the next time piston A executes.

  • If you are using older zwave devices (and some zwave plus devices), these devices may not report physical switch state changes in a timely manner, so you may need to add a refresh in your pistons before checking state for conditions. For triggers or conditions, Zwave Poller should be enabled for these older devices. This is due to ST vs. HE differences in handling of older zwave devices. If you choose Zwave poller, you won't need the refresh commands in you pistons:

  • Version numbers

    • interoperability with the dashboard.webcore.co site requires compatible version numbers for dashboard server interactions. For server interactions webcore reports the webcore version number understood by the dashboard.

    • An HE version number is also in the code for webcore.groovy and webcore-piston.groovy. These are checked that they are the same, but they do not provide an update notification of newer version available (only the webcore version number is checked for the notification)

webCoRE Operations

Your piston is stored in HE (inside the piston). When in the webCoRE dashboard and you want to edit a piston, the webCoRE dashboard requests the code from HE, and when you "save", it stores the code back into HE into the piston.

  • This is why it is good in general for HE to take regular backups on your hub and store off your hub.

webCoRE uses the cloud for webCoRE dashboard operations (when you use the cloud instances of webCoRE dashboard such as dashboard.webcore.co or staging.webcore.co). The webCoRE cloud servers allow:

  • you create/edit/debug pistons (regardless of your location)
  • share, import pistons with others
  • cloud backup and restore of pistons
  • local backup and restore of pistons
  • show the dashboard / tiles of your pistons
  • provides some services such like ifttt (outbound) and email (outbound), functions
  • display of fuelstreams data (storage of fuelstream data is local)

Your pistons execute locally, and if the webCoRE server is down, pistons keep working (unless using one of the above mentioned functions).

staging.webcore.co is the newest / beta of the wc servers (WC team puts updates there before they release them). You generally do not need to use this site unless instructed to, as it may have code that is not fully tested/released.

Piston Backups

  • When you backup your HE Hub, you do have a backup of all apps and settings

  • webCoRE contains the ability to backup pistons locally (to the system you are running your browser on) from the webCoRE dashboard.

    • due to the HE platform limitations on message response size to http requests, you typically have to backup pistons 1 at a time to store locally.

      • specifically HE platform limits responses to (128KB-header sizes) in size, so it may be possible to backup 2 or more smaller pistons, or only 1 large piston in a single request depending on the size of the individual pistons.

It is possible to run the webCoRE servers locally. This is an advanced/complex configuration, and when setup properly removes all webCoRE cloud dependencies - shifting the dependencies to your local deployment. This feature is only available on the HE port of webCoRE.

  • Because your local copy is providing the above server functions, you need to ensure it is kept running, and backed up to ensure no service or data loss of backups of webCoRE pistons.

  • If you want remote access to the webCoRE dashboards, or have multiple locations and want to share pistons across them (when running a local webCoRE server) you have to enable access to the webCoRE server on your local network to the desired locations (VPN, remote desktop access, etc.)

  • It is not required to run the webCoRE servers locally. For information/discussion on running the webCoRE servers locally, see the thread: Patched webCoRE for Hubitat (2018/09/09)

Fuelstreams is a groovy module to store data for piston fuelstream requests locally on the HE. The webCoRE dashboard can display / graph this data.

Release History

Previous updates History / Changes:

Updated Release Updated August 30, 2019

  • update for device optimizations

Updated Release Updated August 25, 2019

  • few bug fixes for color settings
  • more cleanups

Updated August 22, 2019

  • Fix for HSM settings issues
  • Few more optimizations
  • import URLs are added to main modules to make easier updating
  • updated version number to match recent webCoRE release in IDE
  • Normal update procedures should be followed (see below)

Updated August 15, 2019

Significant updates to webCoRE on HE to reduce overhead and improve performance

As always, before upgrade, do take backups of your hub, and store the backup off the hub on your computer. (HE console -> Settings -> Backup and Restore)

It is good practice to reboot your hub prior to upgrading, and to let it settle its internal DB for a few mins, then install the new webCoRE version (HE console -> Settings -> Reboot Hub)

Install the main webCoRE.groovy application last when updating the groovy components (HE console -> Apps Code -> webCoRE Piston; webCoRE Storage then webCoRE (main app).

After upgrading recommendation:

  • go into the HE console, Apps -> webCoRE and then scroll down to hit "done"
    • wait about 1 minute for updates to occur as part of the upgrade
  • open the webCoRE dashboard/IDE in a new browser window
  • THIS SHOULD NOT BE REQUIRED WITH OCT 10 and later updates select a small piston so that db uploads to the IDE will not exceed HE limits. It is possible this small piston may need to be loaded twice (without editing the piston) for all the IDE uploads to complete. This may occur if you have a lot of global variables, variables, or devices. HE logs (HE Console -> Logs) will show if you need to load twice or not.

Improvements

  • much improved performance due to many optimizations - These together can reduce overhead by up to 50%
    • reduction of state usage (HE database)
    • improved locking performance
    • code optimizations to improve performance
  • Added new trigger capability
    • if Device receives attributeValue or if Device gets attributeValue (for momentary devices like buttons)
      • receives / gets triggers have the piston only subscribe to attributeValue (vs all state changes) of the attribute.
  • updated version numbers to match updated webCoRE release

Updated June 27, 2019

  • webCoRE servers updated with HE changes - no longer need to use staging site.
  • updated version numbers to match updated webCoRE release

Updated May 31st, 2019

  • Updates for device capabilities and commands (auto-fill in IDE)
  • fix for physical and programmatic device trigger filters
  • Experimental support for $weather variable based on APIXU data
    • this can be enabled by:
      • installing the storage app
      • entering your apiXU key in the webCoRE app (HE Console -> Apps -> webCoRE -> Settings)
    • data format is apiXU data format (pistons need to be modified to this data layout)

Updated May 12th, 2019

  • Fix for subscription publishing (if you think you hit this issue, pause/resume of the piston, or edit/save (both in webCoRE dashboard) will resolve.
  • Ability to set default notification device used when you call sendPushNotification. This can be the HE mobile app, pushOver or twilio (or any other driver that supports deviceNotification.

Updated May 5th, 2019

  • Ability to execute Rule Machine rules from webCoRE. To use this capability you need to use the webCoRE dashboard site staging.webcore.co when editing pistons that call RM.
  • Improved DST processing
  • improved sunrise/sunset events to avoid double executions
  • cleanups and bug fixes

Updated April 20th, 2019

  • Fix for time schedule subscriptions

Updated April 16th, 2019

  • fix for improper hashing of location ID
  • async http for email requests
  • fix for subscriptions to be properly updated after save of piston code
    • After installing this update, in the Hubitat console, "Apps" -> select each piston and hit done so the updates are applied.
  • update of color code. Note webcore manages hue for colors as a range 0...360
    • webcore automatically scales this to 0..100 range required by color devices that accept hue calls on HE.

Updated April 6th, 2019

  • fix ifttt calls into webCoRE

Update April 4th, 2019

  • fix for access error on field.

Updated April 2, 2019

  • Fixed an issue with editing new pistons.

Updated March 30, 2019

  • Key update is code changes to dramatically reduce state (storage) usage by webcore pistons

    • side effect to be aware of once this new version is installed, you cannot easily go back to an earlier version of webcore on HE due to the state changes / reduction in the piston.
    • piston backups, restores from webcore server work as normal (no changes).

  • Benefits:

    • I have observed startup, and atomicState accesses to be much faster (likely due to having to load/store much less state during startup and piston end)
    • Many operations run faster, especially ones that accessed atomicState and piston startup

  • How to install: (this is consistent with instructions below)

    • Do restart, and backup your HE Hub (download the backup to store off your hub)
    • Install webcore-piston.groovy, then webcore.groovy under "Apps Code"
    • After installed and operating, you should take another backup of your hub and download the backup to store off your hub.

Updated March 23, 2019

  • ifttt events are directed to specific pistons vs. all pistons
    • This change requires you on installation to this March 23 version to pause pistons that subscribe to ifttt events, and then after the groovy code update is complete, unpause them so that the subscriptions are properly updated by the new code. This is done in the webCoRE dashboard, selecting each piston that subscribes to ifttt events / callbacks.
  • time parsing fixes for iso 8601 strings
  • this release has beta / experimental display of local fuel streams in the webcore dashboard. (currently requires staging.webcore.co as dashboard URL)
  • small bug fixes

March 20th changes:

  • Avoid duplicate commands to physical devices
  • Additional time cleanups

March 16th changes:

  • command cleanup and ensure webcore dashboard updates with command changes
  • use this version to avoid issues with recent Hubitat platform changes

March 15th changes:

  • Many time handling and DST changes based on bugs found in original webCoRE

March 14th changes:

  • Global Variables do not wake all pistons on change (only pistons subscribed to that variable)

    • This change requires you on installation to this March 14 version to pause pistons that subscribe to global variables, and then unpause them so that the subscriptions are properly updated by the new code. This is done in the webCoRE dashboard, selecting each piston that subscribes to global variables.
#3

Updated March 20th See original note for fixes.

Updates on hub lockup discussions on HE

2 Likes
#4

I get this warning and even though I cleared the cache and restarted the browser it won't go away.

A newer UI version (v0.3.10a.20190223) is available, please hard reload this web page to get the newest version.

#5

I think for some reason it’s looking at the ST value?

#6

on HE make sure you have updated both webcore.groovy and webcore-piston.groovy.

If you have done this correctly, you can then go into Apps -> Webcore and hit done

The webcore dashboard checks the webcore version number. The code (webcore.groovy and webcore-piston.groovy) check the HE version number (that they match) - this would show up in the HE dashboard under "Logs" if there is a problem.

#7

Not working.... I must be doing something wrong.

#9

i dont care webcore can suck it

#10

Updated version today with a few fixes.

See note 1 in this thread

#11

for anyone using local fuel streams on HE,

There are fixes pending testing on the staging.webcore.co dashboard site. (do note these are unreleased and under testing) that enable display of local fuel streams in the webcore dashboard -> "FuelStreams"

#12

You can just ignore this.

#13

It normally should only show up once if the webcore servers are updated.

If you are running a local copy of the webcore servers, this will continue to come up until you update your copy of the webcore server.

Generally older server and newer groovy files should work, however some features may be missing (if this message continues to come up).

As stated in the instructions in message 1 here, when updating (potentially groovy and local webcore server) it is good to update one thing at a time, so you can more easily checkout that everything is good in your environment (ie do groovy first, run for a while, then do your local webcore server).

1 Like
#14

New Update posted . March 30, 2019

  • Key update is code changes to dramatically reduce state (storage) usage by webcore pistons

    • side effect to be aware of once this new version is installed, you cannot easily go back to an earlier version of webcore on HE due to the state changes / reduction in the piston.

  • Benefits:

    • I have observed startup, and atomicState accesses to be much faster (likely due to having to load/store much less state during startup and piston end)

    • Many operations run faster, especially ones that accessed atomicState and piston startup

How to install:

  • Do restart, and backup your HE Hub (download the backup to store off your hub)
  • Install webcore-piston.groovy, then webcore.groovy under "Apps Code"
3 Likes
#15

If we are updating to this version from an old version (pre 2019) do we have to do all the installation workarounds listed in the previous versions also? Do we have to step through and install all the previous versions first? Or can we jump right to this version?

#16

You should be able to go straight to it.

#18

Hi, I just try to install : webcore-piston.groovy in this link : webCoRE/webcore-piston.groovy at hubitat-patches · jp0550/webCoRE · GitHub

When I hit Save on HE, it say : Metadata Error: Parent App 'ady624:webCoRE' not found on line 301
I can not save this app ! Please help. Thanks.

#19

Just like it says, you need to install the webcore app code 1st then piston.

1 Like
#20

Now I need to install " webcore.groovy " 1st , is that right ? Thanks.

#21

Update April 2, 2019

  • Fixes a bug that editing new pistons did not open the dashboard edit screen properly.

Thx @Royski for the report on finding this.

2 Likes
#22

When I open the Dashboard Webcore on HE, and hit new piston, nothings happen! Did I miss something?
Thanks a lot for your help.