[Sneak Peek] Hubitat Package Manager

I'm probably going to need other devs to help out with this one and I'm not even 100% sure I'm going to be able to make this work, but something I'm currently toying around with. I think the way you have to install apps and drivers is too complex for less technical folks. At the same time, I think many of these packages out there are the reason HE is great. The stuff that comes with HE is nice, but there needs to be an easier way to add apps and drivers from the community. This is pretty far away from even being an alpha release, but I think the concept will be really useful to making it easier to install apps and drivers. Any feedback?

24 Likes

Personally, I don't find it necessary but I think it a good option for your target audience. The only thing that jumps out at me is the "URL of the package manifest" which I think will confuse less technical noobs. Getting code sharers to advertise the url with the same terminology will be key in getting around that confusion, IMO.

Thanks for thinking along this line as I've seen comments in several topics lately that request simplification for the less technical.

1 Like

Good idea, Also want to point out that some developers (like Andy @Cobra) provide comprehensive install instructions with apps/drivers, including email notifications of updates.

3 Likes

It’s certainly a welcome idea to have a way to see if the random groovy script I copied into my hub has been updated or not...half of the scripts out there don’t even have version numbers attached.

I think a manifest file is a great idea because it has the potential to allow the driver, parent and child apps to all be “packaged” even if there is no way to break code up into multiple files.

What API are you using to add apps and drivers to the hubitat?

That was kind of the idea. I have one app in particular I wrote (BOND Home Integration) that comes with an app and 10 drivers. That's a lot to install. Or something like Ecobee Suite which has 13 apps and 2 drivers... that's a lot to install!

Unfortunately there is no API that I've found to do it. So basically I'm sending HTTP requests that mimic you going to the Apps Code or Drivers Code pages and uploading an app.

Like I said, still a lot of work to do. The thought is though these more complicated apps can be installed by loading a single file and then you have one screen you can go to to check for updates for all your packages at once.

Also I can do other fun things. Like I have a setting in there where the developer can say what version of HE is required. If I use a new API that was added in 2.1.9, let's say, well I want to give you a nice friendly error message if you try to install it on 2.1.8 that you need to upgrade.

I plan to let you mark certain things as optional. Like for Ecobee Suite, the main app would be required, but let the user choose which of the 12 child apps they care about, etc.

As a said, a lot of work to do, but before I invest a week or two in this I wanted to see if people thought it would be useful.

3 Likes

I don't agree that it's only for "less technical folks". I've been in the industry my whole life (37 years, now retired) and playing with Smart Home since X10 days starting in 1981.

I find that Hubitat is amazing extendable, but that proves that this is needed if only for the "Update" function. That would be amazing. I know there is a "external" checker, but I perfer to try and keep it "native".

Love to Alpha test it if you want.

(Will it work for drivers?)

Alan

1 Like

This looks very promising. Looking forward to seeing it evolve.

Well, there is certainly value. Just a week ago the iPhone WiFi presence updated and introduced a bug, and the developer was quick to fix it but had to respond to half a dozen people who had upgraded and tripped the bug.

The approach of banging the http api is clever, but not without its risks. I think the biggest risk to something like this is that your database gets corrupted and people have to clean things up manually... but depending on what apis you’re banging on you might well be able to sort that out too. Deleting is risky...if someone has a lot of time invested in some app config and you nuke the app, a lot of human life just got evaporated.

As for doing work which no one ends up using, I think your biggest risk is that you’re veering a bit from the hubitat green path. Hubitat corporate could well be workin on something like this, and if they do it would be a bad idea to continue using what you built regardless of when than happened to arrive.

I'm just guessing what you're going to need from a candidate package based off what little you've shared. Each package will need a manifest of some sort, and each file in the package will need a magic comment as well. The manifest allows the package manager to install all of the required files, and the magic comment on each file allows you to traverse backwards from file to manifest to check for upgrades.

If that's true, I guess my request would be to keep the requirements simple enough that they don't add to the mental overhead of the project I'm working on. Providing a script which can do the legwork of a release is a wonderful gift as well.

I'm a big proponent of SemVer if you need version numbers.

If you end up needing a magic comment in each file, from the bottom of my soul I ask you to avoid this pattern:

/*** START OF MAGIC PACKAGE MANAGER COMMENT MAGIC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
*** version=1234
*** package=awesome.package
*** package_url=url://package.awesome/packages/awesome/package.awesome
*** package_icon_url=url://package.awesome/icons/packages/awesome/1234/package.awesome/icons/icon3_3x4.png
*** END OF MAGIC PACKAGE MANAGER COMMENT MAGIC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
***/

Instead, keeping it to one line and straightforward would be easy to script against and not be visually distracting :

// hpm version=12 manifest=url://package.awesome/manifest.json

I'm being a bit hyperbolic here on purpose to balance out my overzealous design direction ...but you asked for opinions so here it is! :slight_smile:

Yup

Agreed. Unfortunately I haven't found a better way at this point. If anyone has suggestions...

It is, but that's where "are you sure?" type messages come up :slight_smile:

I agree, this is the biggest risk. If the HE team were to say "we're working on it" I'd probably halt development. The problem is, they don't (and don't want to) publish a roadmap so makes it a sticky situation... I'd much prefer this to be built into HE. I think it's definitely the proper path. At this point all I've heard is they want to do something like this, but haven't started. So maybe they can use this manifest idea as the foundation for what they build in?? I'd love for the HE team to chime in if they would prefer me not to walk down this path.

That'd be my goal if I take this as far as I hope to. I'd love to just build a little program that generates a manifest.

Agreed

The plan is not to need it. The idea is to just provide a manifest file that you install from which has all the needed details. Here was something like what I'm thinking, https://raw.githubusercontent.com/dcmeglio/hubitat-bond/master/packageManifest.json

Love it. Happy to help however it's needed.

@bravenel or someone else from HEHQ, care to comment on this?

I got pretty far on this today. You can install packages, uninstall, modify existing ones (add/remove optional components), and execute updates. When installing a package, if it requires OAuth it will automatically enable it for you without having to go to the editor. You can specify a minimum HE version that is required. Each app and driver in the package can be declared as optional or required. There is a TON of error handling that needs to be added (I have to come up with a way to do rollbacks which will be a real challenge). I'd also like to move some of the upgrade process to async so I can provide realtime feedback in the UI as steps are complete.

Making really good progress for about 10 hours of work total!

A couple more screenshots:

6 Likes

This is a nice innovation.

We certainly have had many conversations about this subject, and may introduce some tools at some point in the future in this general area. But that doesn't mean you shouldn't go forward with this.

At any point in time new things may be introduced that effectively obsolete something people are using, be it a custom app, driver, or whatever. But, so what? If what you use today works, just because there might in the future be another way to do it would not mean that you'd have to change what you're doing. We do make an effort not to change things in such a way that would blow things out of the water.

I should also say that this area is not a top priority for us right now.

13 Likes

about 10 hours of work total!

You are a machine, wow

2 Likes

Thanks for the reply Bruce. Agreed, you guys can make anything obsolete. Just wanted to make sure you weren't going to say "It's #1 on our list" because then I'd just be wasting my time and creating a mess for the community.

One question you might be able to help with. There is no way to validate that an app or driver is "valid" without installing it, is there? Meaning there is no endpoint that would let me compile it, ensure it would install, but without actually doing so? I only ask because that would make the need for a rollback a lot less likely...

Btw, if any other devs out there would like to take a look, send me your github username. I fully intend to make this public and open, but first I want to get it working. Also I know it will need a LOT of testing as something like this really needs to be near flawless to put into the wild!

@dman2306, I'd be more than happy to help test and give feedback. GitHub username is the same as it is here, bptworld.

Thanks!

3 Likes

Added, didn't realize that private repos on a free account only let me add 3 people, so if a bunch are interested I'll just make it public.

Thanks! I used your packageManifest as a guide and created a new one for one of my apps to test. Very easy to do and well laid out. The only question I have is on the packageId, App Id's and Driver Id's. Do you recommend a program to create this string of digits? Will each dev get some sort of identifier so no one else can create the same set of digits? I simply changed the first set of numbers to my zip code to test.

Testing went very well. I installed a package, then modified it and then uninstalled it.
Mind blown! This is going to make things a lot easier for everyone!

I look forward to moving all of my apps/drivers to this down the road.

Also, feel free to PM me with anything you like me to do, test, etc. :wink:

4 Likes

They're GUIDs/UUIDs (Globally/Universally Unique Identifiers). In theory, they're unique in the universe unless you purposely make a duplicate. I plan to make a tool to generate the manifests, but in the meantime https://guidgenerator.com/ works.

To try to make this even easier on end users, I just introduced the concept of a repository. That is a place where you can list out all of your packages so the user can just choose from a list. So how do people find out about your repository you might ask?

@aaron and I discussed this and came up with this.

There is a 3 tier structure:

The tricky part is #1. How do you get your repository in the list? Well the best we came up with was sharing access to that github repo with a trusted set of HE community members. I really don't want to be a gatekeeper to who can release packages and who can't. This was the best we came up with for now. If anyone has any suggestions, I'm all ears!

Anyway this is the install screen now:

There is still a bunch to test and cleanup before I consider this ready for prime time (I already identified a few bugs). I've cleaned up the code enough to where I'm no longer embarassed by it, so I just made it public. You can find it here GitHub - dcmeglio/hubitat-packagemanager

Anyone who wants to look at the issues list and submit a PR, I'm certainly open to help :slight_smile:

5 Likes

The app is really cool... can't wait to see how it turns out.

For anyone interested in contributing their apps to use this framework, I added documentation on how to create a packageManifest, a repository, and how to get your repository added to the list. You can find this info at hubitat-packagemanager/README.md at master · dcmeglio/hubitat-packagemanager · GitHub

Hopefully people buy into this, it'll be kind useless if only 1-2 devs use it :slight_smile:

6 Likes