Thoughts on an enhancement to Hubitat Package Manager

jpage4500 · October 28, 2020, 2:07pm

I've been using HE for almost a year now and things are going great! Coming from Wink the biggest frustration I had was not knowing if or when product XYZ would ever be supported by the platform and with Hubitat and the amazing developer community I've integrated products I didn't even consider integrating with Wink..

Orbit Hose timer - check
Google Nest Thermostats - check
Blink XT cameras - check

Of course with all of these various apps and drivers there's going to be a need to update. I have the Hubitat Package Manager and am a big fan but of course it only works with the apps that support it. For all others, I'm still manually updating the drivers 1 by 1. While I'm tempted to say let's just get all apps supported in the Package Manager I'm wondering if it's really necessary to require these drivers to change at all?

I've noticed for most drivers I'm using, updating simply requires me to hit the "Import" button and the URL is already pre-filled. I just hit Import again and then Save - that's it.

That led me to wonder.. can't someone write a script/app which looks at all of your drivers and compares the source to the version that's pointed to in the importUrl field and notify me if it changed?

I even thought about doing this in my dashboard app even though it's more of a utility thing that doesn't really belong there.

syepes · October 28, 2020, 3:24pm

I am doing some similar with my apps/drivers just to check the version no updating..
But even if we create an app that's going to check this not all code out there is using the importUrl.

I have recently started using HPM and it's very nice, what would really be great is to have something OOTB/Native from the HE.

marktheknife · October 28, 2020, 4:02pm

I agree it would be nice but they really open up a can of worms the more they semi-officialize (probably not a word but ) the process of adding community code to the hub.

It can be hard enough for them to provide support for all the hub’s native features.

rsjaffe · October 28, 2020, 4:03pm

The problem with that idea is that the Hubitat folks would have to approve any packages that were added. Imagine if they didn't and someone registered a malicious package?

syepes · October 28, 2020, 4:12pm

Well yes and no, it's just like what we have right now and other package managers: NPM, AUR.. it's up to the user /responsabily of knowing what he is installing..
It jus need to be written down so end users are informed.

marktheknife · October 28, 2020, 4:27pm

I'm not sure I follow. Hubitat package manager is a community app that automates the installation and updating of other community apps/drivers.

If Hubitat were to further facilitate the installation of community code with a built-in package manager, it could create an expectation among users that it's Hubitat's job to support whatever goes wrong with the installation/update processes for that code, or even problems with the actually driver/app code. Worst case, as @rsjaffe points out, is that code available in that built-in package manager is perceived as having some stamp of approval from Hubitat staff, but turns out to be malicious in some way.

They can add disclaimers, but people will ignore them, and will still open support tickets or take to the forums to bemoan their plight if something goes wrong.

jpage4500 · October 28, 2020, 5:03pm

I think HPM (I like having that acryonmn) is fine as a community app. I also think that once a user discovers it -- it makes finding and installing drivers easier than searching through the forums.

My only note is -- and maybe this isn't a problem for others -- is that only 3 or 4 of my apps/drivers are listed in HPM.

All the rest that I can see require manually updating. While that's fine too.. I doubt some of them even are updated very often - I did notice at least in my case that a lot of them specified an importURL field in the driver itself.

At least for the drivers that set this field -- and are not in HPM -- it seems possible to have an app check for updates to these. I guess maybe in my ideal world HPM could do this -- just as a way to cover a lot more apps & drivers that wouldn't get checked otherwise. Not that they should do it though as maybe it would discourage devs from official adding to it.

But, it only matters though if the majority of drivers even set that importUrl field to begin with like in my case.

csteele · October 28, 2020, 5:20pm

Personally I think importUrl field is not valuable. It does not yield the #1 need of HPM, which is the correct version. Version is not a required value in App or Driver code.

The critical missing element is a FAST means to check versions. importUrl points at the source, which may or may not include a version. HPM would have to download EACH source file and scan it looking for a version that may not exist.

I don't think that's a burden any of us want our hub's attempting.

HPM has an option to Add a Custom Repository but, again, that's looking primarily for version numbers. It would be a Community effort to gather the current versions of non-HPM-supported code and then build out a versioning mechanism...

Put another way, HPM has two complementary sides.. version detection and code import. Detection would be a benefit BUT it would be a continuous effort because the 'owners' of the code would continue to not update the version file. That is, after all, 90% of the commitment made when adding a repository 'officially'.

lewis.heidrick · October 28, 2020, 5:38pm

Just put the raw gethub url in the import url and be done with it for any that aren't in hpm. Can check it every so often or if you start having issues or want some feature the new version provides.

jpage4500 · October 28, 2020, 5:51pm

I don't disagree with any of this. And I wouldn't want the hub to do this automatically either as that would be a waste of resources.

But, most of these drivers are only a few (k) -- and if this update check was done manually I don't think it's too much either.

Anyway, for me it's more about just an easier way to surface if there's any updates available.

I do this in my app today for hub updates. It's nice IMO because I rarely view the hub's webpage anymore but like staying on top of the latest versions.

Anyway, I'm def not capable of writing this as a HE app but if I did it in an Android app or via bash script it'd be something like this:

- user hits 'check for updates'
- look at all of the drivers listed in 'drivers code' and apps code (ie: http://<HUB IP>/driver/list)
- for each one:
    -- check if there's an importUrl field defined
     - download the file pointed to by importUrl
     - compare with the current driver's source
     - if different add to a list of modified drivers
- display the list of modified drivers so user can choose which ones to update
- for each driver to update:
    -- TODO: figure out how to submit the updated source.. haven't gotten that far yet!

csteele · October 28, 2020, 6:12pm

Before importUrl existed, or was commonly used, someone added the import URL to the top of their code. I saw it and the light went on and I added it to all of my code. I also rabble-roused as many as I could to include... and since it's a once per lifetime addition, I seemed to have great success. (Ok, others did this too but I'm happy to pretend it was all my effort ) Then importUrl came along and again, because it was simple and one off, it was mostly included.

Then, for me, Cobra's idea of including a weekly check against a file for an update occurred and while I eventually deviated in the file format, for my own efficiency in updating the file, I still, to this day include the feature. Thus I must maintain TWO files containing the same info, but in two formats. Maybe someday that burden will grow to be worth the effort to change all my code to use a single file, but that day has not come, yet.

For me, I feel that a release of updated code is dominated by the version updating effort. I won't release tiny changes because the 5 mins it takes to update the version files (all of them) encourages ME to delay/defer.

Screen Shot 2020-10-28 at 11.08.40 AM

Which is easier to update/maintain??

{
  "packageName": "Auto_Off",
  "author": "csteele",
  "dateReleased": "2020-10-11",
  "apps": [
    {
      "id": "b03472ad-a305-4544-8e76-1d5b39e750b0",
      "name": "Auto_Off",
      "namespace": "csteele",
      "location": "https://raw.githubusercontent.com/csteele-PD/Hubitat-public/master/Auto_Off/Auto_Off.groovy",
      "version": "1.0.1",
      "oauth": false,
      "primary": false,
      "required": true
    },
    {
      "id": "dd3ec702-7bee-4a46-aad4-b9a53145d9ec",
      "name": "Auto_Off device",
      "namespace": "csteele",
      "location": "https://raw.githubusercontent.com/csteele-PD/Hubitat-public/master/Auto_Off/Auto_Off_c.groovy",
      "version": "1.0.3",
      "oauth": false,
      "primary": false,
      "required": true
    },
    {
      "id": "887bf9e2-41ce-483b-a9d7-121a26d76eb1",
      "name": "Auto_Off poll",
      "namespace": "csteele",
      "location": "https://raw.githubusercontent.com/csteele-PD/Hubitat-public/master/Auto_Off/Auto_Off_o.groovy",
      "version": "1.0.1",
      "oauth": false,
      "primary": false,
      "required": true
    }
  ]
}

OR

{
        "author": "C Steele",
        "comment": "",
        "copyright": " &#9400; 2020 cSteele",
         "application": {
               "Auto_Off":  {"ver": "1.0.1", "updated": "10/14/2020"},
               "Auto_Off_c":  {"ver": "1.0.3", "updated": "10/14/2020"},
               "Auto_Off_o":  {"ver": "1.0.1", "updated": "10/14/2020"}
         },
         "driver": {
         }
}

Additionally, all my code has:

public static String version() { return "v1.0.2" }

which is probably one of the rare formats... defining it as a method.

dman2306 · October 28, 2020, 11:18pm

What it really comes down to is - what problem are you trying to solve? When I was bored and thinking about HPM, I identified 3 problems I wanted to tackle:

Discovery- there was no good way other than searching on these forums to find new packages
Installation - some "packages" are complex. Ecobee Suite is a great example. It's an awesome app, but one that requires installing 2 drivers, about a dozen apps, and requiring the user to manually enable OAuth. As a developer myself, I get lots of support requests of "it doesn't work" only to find out it's because they didn't install all the files they needed
Upgrades - Other than manually searching Github (assuming the author even puts it in Github, some drivers are only found pasted in posts on the forums!) there was no good way to know updates are available.

I thought about the idea you're discussing here, however, it only addresses #3 so it didn't solve my problem statement. That's why I went in a different direction. Even for #3 I found it a partial solution. For example, imagine you have now moved some of your app code into a separate child app. How would, looking just at importUrl, it know that there is now another app that needs to be installed that isn't already? Upgrades, I realized, get a bit more complex than just "update this one groovy file." File manager made it even more complex with things like Smartly and Hubigraphs requiring you to install JS/CSS files via File manager.

I wish some kind of package management was a first-class citizen in HE. I think it would have a much higher adoption rate (both on the user and developer sides). But I don't think it's happening any time soon. When I started building HPM I asked the HE team if something like this was in the cards any time soon (because I didn't want to waste my time if it was!) and they said no, though it is at least on their radar.

ourmessages · October 29, 2020, 12:10am

I appreciate HPM, without community apps HE wouldn’t be much of itself. It’s value and usefulness is 85% because of community apps which without the free contributions of talented persons HE wouldn’t have much appeal or usefulness to me. That said, it really should have a built in way to find apps. But developers have come and gone from the community and others have put their apps on their own sites ( which they can do ). Unfortunately these are the issues we all face when something is mainly supported by other users of the platform vice a tighter ecosystem where the company controls the entire process. Both user and company controlled ecosystems have their pros and cons but with user supported ( as I saw with other systems ) supporters come and go and then apps get abandoned and you either find a fix or move on to the next “latest and greatest” platform. Everything cost something. With HE you pay in your time to create functionality. With ( like Apple ) you pay in $$ to have a system that just works. Unfortunately with HA/HC there is just no one option to get this lightbulb I want to Control this dishwasher I want, so you have to compromise or devise ways to make interoperability happen.
Knowing what / where / how in regard to apps / drivers would be a great step toward removing the self-service aspect.

jpage4500 · October 29, 2020, 2:49pm

I realized this post wasn't named very well.. I was never looking for an 'alternative' to HPM but just a way to pull in support for all of the apps and drivers that aren't using HPM.

I haven't written a single HE driver so I'm an outsider looking in on this one.. just happened to notice that a lot of the drivers I'm using aren't listed/supported in HPM. However, they do have an importUrl listed which to me seemed like somewhat of a standard (and maybe it's not.. again, just looking at the drivers I'm using but it'd be nice to know what the majority of HE drivers do -- HPM/importUrl/internal version check/nothing?)

What I can relate to as a long-time developer is that versioning can be a PITA.. I go to great lengths myself to keep things as automated and simple as I can.. for instance in my Dashboad app the builds, version number and releases are all automated. I add some features.. test them out and merge them in and the rest is handled automatically.

Anyway, long-story-short - I was just trying to think of a way to accommodate more drivers.. seems like there is at least a possible way to do this. Certainly for the drivers that are on github as versioning is built-in..

neonturbo · October 29, 2020, 3:26pm

I know what you mean. I wish I could put the Import URL in Hubitat Package Manager, and have it check that somehow. In reality, I know why that won't work, but I can wish, can't I?

You should be able to rename it. If not, I can change it for you.

danabw · October 29, 2020, 3:38pm

Yup...click on the pencil next to the name, @jpage4500:

csteele · October 29, 2020, 3:45pm

All of us can get in on the fun...

dman2306 · October 30, 2020, 1:12am

Someone built what you’re suggesting. I just don’t recall the link. I think it was a python or node app. Can’t remember.