Proposal for a Library Dependency System in HPM

⚙️ Custom Apps and Drivers Hubitat Package Manager
1

Proposal for a Library Dependency System in Hubitat Package Manager (HPM)

This post outlines a proposal for a new library dependency system in Hubitat Package Manager. The goal is to create a more robust and automated way to manage shared code (libraries) across the Hubitat community, addressing the limitations of the current "Bundles" system while leveraging the existing infrastructure.

The Problem: Managing Shared Code in Hubitat

Currently, managing shared code in Hubitat, and in HPM is challenging:

  • Manual Process with Bundles: The existing method for distributing libraries in HPM is through Bundles. This process is manual, requiring developers to create .zip files. It does not provide a seamless way to manage updates or dependencies.
  • Code Duplication and Poor Maintenance: Without a proper dependency management system, developers often resort to copying and pasting shared code. This happens both across the community and within a developer's own collection of drivers. A complex driver system with multiple components can lead to significant internal code duplication, making maintenance difficult and error-prone. On a community level, this leads to inconsistencies, with a prime example being logging frameworks where nearly every developer has their own implementation.
  • No Automatic Updates for Libraries: When a developer updates a library, there is no automated way for HPM to detect and install the update for all the drivers and apps that use it.

The Proposal: A True Library Dependency System

I propose the creation of a proper library dependency system within HPM that would automate the installation and updating of shared libraries. This system would be based on extending the existing repository.json format that developers already use.

Key Features

  • Decentralized Library Definitions: Developers would define the libraries they publish within their own repository.json file, alongside their existing packages.
  • Dependency Declaration: Driver and app developers would declare their library dependencies in their manifest.json file.
  • Automatic Resolution and Installation: When a user installs or updates a package, HPM would automatically resolve, download, and install the required libraries.
  • Automatic Updates: HPM would be able to check for and install updates to libraries, just as it does for apps and drivers.

Proposed Technical Implementation

1. repository.json Modifications

Developers would add a new libraries array to their existing repository.json file. This array would live at the same level as the packages array and would define the libraries they wish to make available.

Example repository.json:

{
    "author": "A. Developer",
    "gitHubUrl": "https://github.com/a-developer",
    "packages": [
        {
            "id": "f9e1f5b0-c5e5-4f3c-b8f9-2e1b8a0b1b9e",
            "name": "Sample Weather Driver",
            "category": "Integrations",
            "location": "https://raw.githubusercontent.com/a-developer/hubitat-weather-driver/main/manifest.json",
            "description": "A driver for a sample weather device."
        }
    ],
    "libraries": [
        {
            "id": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
            "name": "CommonUtils",
            "namespace": "a.developer.libraries.utils",
            "version": "1.0.0",
            "location": "https://raw.githubusercontent.com/a-developer/hubitat-common-utils/main/CommonUtils.groovy",
            "description": "A library of common utility functions."
        }
    ]
}

2. manifest.json Modifications

A new, optional, libraryDependencies array would be added to the packageManifest.json format. To ensure a unique reference, developers would specify both the library name and its namespace.

{
    "packageName": "Sample Weather Driver",
    // ... other manifest properties
    "drivers": [
        // ... driver definitions
    ],
    "libraryDependencies": [
        {
            "name": "CommonUtils",
            "namespace": "a.developer.libraries.utils"
        }
    ]
}

Example Workflow

  • Library Developer:
    1. Creates a new utility library.
    2. Adds a libraries section to their repository.json file and pushes the change to their GitHub.
  • Driver Developer:
    1. In their driver's manifest.json, adds a dependency on the library by specifying its name and namespace.
    2. In their driver's code, uses #include a.developer.libraries.utils.CommonUtils.
  • End User:
    1. Installs the driver via HPM.
    2. HPM automatically looks up CommonUtils using its unique namespace and name combination, finds its location, and installs it.
    3. When the library developer updates CommonUtils, HPM notifies the user of the update during its next check.
      Edit: That wouldn't work, libraries would have to be versioned and tagged. Drivers would declare a specific, immutable version.

Benefits

  • Reduces Code Duplication: Encourages the use of shared, community-vetted libraries.
  • Improves Code Quality: Allows for the creation of standardized libraries for common tasks like logging, data parsing, and device communication.
  • Simplifies Development: Developers can focus on their app or driver's unique logic instead of reinventing the wheel.
  • Enhances User Experience: Users get a more robust and reliable system with automatic updates for all components.
  • Supports Both Public and "Private" Libraries: A library dependency doesn't necessarily need to be for community-wide use, like a logging library. It can also be a private utility library that a developer uses to reduce code duplication across their own complex driver system, but which is not intended for general consumption by other developers.
  • Decentralized and Scalable: Developers can publish and update libraries without needing to submit a PR to a central repository for every version change, making the system more scalable and reducing the maintenance burden on the HPM team.

Initial Scope and Future Considerations

For the initial implementation, I propose that we do not support multiple versions of the same library. New versions of a library would be expected to be backward-compatible, if they intend do be widely used. This simplifies the initial development effort and aligns with the current HPM philosophy.

This proposal aims to start a conversation about improving library management in the Hubitat ecosystem. I believe that a proper dependency system would be a significant step forward for both developers and users.

4 Likes
2

I would propose that Libraries be abolished. "dll hell" or "dependency hell" or "library hell", the meme is all over the software developer world. I completely see the need of Developers to have a library or ten, but why foist that upon the poor public? Want to share libraries? Then share them but compile them in prior to release.

Hubitat uses libraries themselves, but you don't see them foisting that burden on their customers.

The HPM Manifest allows for extension. Anything HPM doesn't need explicitly, it ignores. You can add key:value pairs for anything and then process it elsewhere.

7 Likes
3

Proposal seconded

  • they aren’t true libraries, just code that the platform inserts into the driver or app script at compile time AFAIK
  • there are plenty of traditional tools to do this however Claude Code demonstrates how code can be post processed / assembled with a simple prompt
1 Like
4

Jinx. Came here to say same, but now curious how this convo plays out, especially given (as I suspect) such a limited target audience.

1 Like
5

Hi @csteele,

Thank you for the feedback. I understand your concerns about dependency complexity, but I believe this proposal is fundamentally different. It's a one-level dependency system—libraries cannot depend on other libraries—which prevents the nested complexity that causes "dependency hell" in other library systems.

The approach of pre-compiling everything into a single file adds friction to development. AFAIK it is only used by very few developers. Every code change requires a build step before testing.
Another aspect is that it discourages sharing high-quality, reusable code across the community.
A modular approach could raise the quality of community drivers and apps.

That said, a well-designed system would be transparent to end-users. They would simply install a driver and it would just work—no need to understand the library system at all. And this would remain optional; developers who prefer single-file drivers could continue that way unchanged.

I understand that there are some challenges, and that given your concerns, you probably won't implement this feature, and I respect that decision. Thank you for maintaining such a critical tool for the community.

1 Like
6

I don't see how that is true at all, Create a repository of libraries and anyone can use a copy. That's pretty basic Github, in my view.

"We" have a HubitatCommunity repository already:

with 49+ repos already, It's pretty easy to add yet another.

Obviously collecting Library code is the low-labor portion of the challenge. Getting people to even know such a thing exists and then slogging through the documentation for each is a much larger task. One that would be true independent of the storage location.

HPM would help with none of that, as proposed. The math just doesn't work, in my opinion. Saving One Developer N minutes per commit, is then offset by hundreds of users deep in 'library hell' because of a overlapping name or accidental failure to be backwards compatible. I really feel the problems of a Library System don't get cured just because HPM gets an ability to save library code.

7 Likes
7

Hi @csteele,

Thank you for the continued discussion. You raise valid points and I appreciate that.

First, the correction: You're absolutely right that automatic library updates wouldn't work as I described. Libraries would need to be versioned and tagged. Drivers/Apps would declare a specific, immutable version of a library they depend on. This is a critical amendment to the proposal.

On library dependency hell: Our development experiences were probably quite different. In over 33 years as developer and architect of large backend systems—25+ of them in Java—adding library dependencies like Apache Commons and similar frameworks was not only standard practice but hugely beneficial and pain-free. I've never experienced "library hell" in those decades. Yes, there were occasional version conflicts, but far from catastrophic. The benefits far outweighed the downsides.

Hubitat drivers and apps are vastly simpler than large enterprise systems, so managing a handful of well-designed libraries should be even easier and more problem-free.

On pre-processing: Beyond the development/test cycle friction I mentioned, there's another significant issue: you spend 100% of your development time working with files that differ from what's in git. To keep them in sync with the repository, they must be processed. This creates a constant disconnect between your working code and your source control.

On code sharing: Copying-and-pasting code that evolves as a sharing strategy has never worked and never will. A proper library system is the only sustainable approach to community code reuse.

I understand your concerns about the complexity and the risk-benefit calculation. We simply weigh the pros and cons differently based on our different experiences, and that is fine. I respect your decision on this feature.

1 Like
8

As long as the owner of the shared source code you are happy to test all the uses of the shared code before releasing a change... Or wait for the users of the library to do their testing... Or they are willing to accept that risk of a breaking change being made....

I'm all for sharing code and making it easy for that to happen, but it does come at a cost...

Personally I would be happy to reference a relatively static and common set of code that changed infrequently. i,e, something that should become part of the platform (anything by @thebearmay :wink: ).

5 Likes
9

Isn't that what we do now? Use libraries that are whitelisted by the platform? :slightly_smiling_face:

And periodically, Hubitat adds additional Groovy libraries to that list.

I think my biggest issue with this current proposal is versioning and maintenance, especially the latter.

5 Likes
10

That's essentially my point as well.... Managing changes is the biggest issue IMO

2 Likes
11

But that doesn't mean we should stop it.... stable and useful code like I tongue-in-cheek referenced by including @thebearmay is a good example.... there are some useful and fundamental utilities that would not need to change often... Which would not need to fall into the category of needing to manage changes on a regular basis....

I feel it is more a case of buyer beware for the developer choosing to use a common library, just like in using Community drivers or apps

2 Likes
12

A few thougs on the point of this discussion.

First Bundle Versioning is already present in HPM. It was added a few versions ago when i was working on the Beta Code. Updating bundles by version is possible today. I think i have a bug to work on in reguards to if beta and the current version are the same, but the point is Bundles are versioned.

There is absolutely nothing preventing someone from creating bundled package of code and publishing code it as a library today so that it can be used by the comminity and developer if they want to. I use libraries in my integration currently. My manifest has a bundle that contains 8 files that are libraries for my Govee Integration. I use it extensively to standardize my code across all of my drivers and such in the integration so i only touch the bundled code and minmize my interactions with each driver. If someone actually asked me to create a simple bundle with just the Library files i could do so in short order and publish it easily.

Keeping it updated is a different matter though. I would not be willing to validate someone elses code before publishing updates. I also don't care to document every change since my focus is on my integrations or items i have choosen to take part of and not others. This leads into what @sburke781 stated as it would end up being a risky endevour.

Reguardless, I actually don't see a blocker to a developer releasing a Library Package today if they wanted to. It would just be a bundle file with only Library files in it. The problem is getting folks to actually use it and then the community to know they need to install it. Maybe the developer using the bundle could just include it in their Manifest, but behond that i don't see a blocker from this functining today.

4 Likes
13

Exactly. I have in mind infrequent changes to relatively static, common code. When a library does change, it would be released as a new version with detailed release notes. Existing drivers referencing an older version would continue to work unchanged—no automatic updates. Developers would explicitly choose if and when to upgrade, just as we do with any library, framework, or firmware in traditional software development.

Regarding versioning and maintenance concerns—I completely agree these are important. But as with any tool, no one would be forced to use libraries. The community could develop shared libraries collectively, individual developers could create their own, and others would adopt them if they find value. Otherwise, they simply ignore them. It's the same model that works for big projects in Java: teams choose between Apache Commons, Google Guava, or developing their own utilities.

This flexibility also addresses the use case that @mavrrick58 mentioned. Consider a parent/child driver system where common methods are duplicated across multiple drivers. A "private" library—one intended only for that developer's own use—would eliminate that duplication and reduce maintenance burden and bugs. No one else needs to know it exists or use it. It's purely for that developer's benefit.
I do have to learn more about bundles, I've never used them, so I cannot express an opinion about their pros and cons right now.

The beauty of this approach is that it's entirely optional. Developers who prefer monolithic, self-contained drivers can continue that way. Those who want to embrace modularity and code reuse have that option, with community or private libraries. Both approaches coexist without conflict.

14

I genuinely appreciate your enthusiasm and contributions towards improving practices around here.... You are doing good stuff... :slight_smile:

But... the dev's here, including me, as I would expect everywhere, can be flighty, dipping in and out of their engagement with the Community / platform and practices encouraged by the Community, like you are trying to instigate. In short, unless there is some kind of gold on offer, you will likely feel like you are herding cats trying to get others to adopt any kind of convention that will only succeed when people adopt it consistently... I know HPM is an exception, but it is special :slight_smile:

I hope this makes sense....

2 Likes
15

Thinking it some more, I'm not sure that is entirely true... At least from the point-of-view of the user....

Depending on how it would work... If two HPM "packages" reference the same common library and one of them is updated, then that would presumably trigger the download of the latest version of the library, potentially resulting in a change in behaviour for the other HPM package...

I could be wrong....

EDIT - That is not a criticism of this suggestion per see.... more a likely behaviour of the library feature...

16

You are complete right. Libraries or any code on Hubitat isn't version aware really. HPM manages version in it's own app metadata. It isn't done by the hub. You would almost need to modifiy the way HPM manages libraries to include version numbers in the naming when being installed. That could get messy real fast.

The biggest concern with this is getting folks to actually use it. The beta feature has been around for a very long time from what I could tell. It appears though that for the most part it hasn't been used. I don't know if the enhancements to the Beta features I added are really getting any use be anyone beyond me and jtp10181 . Since I wanted to use them, I took the extra steps to enhance it to what I felt made it more useable for me and the folks I support.

To that end if you feel strongly about a feature you can always work with csteel on adding a feature yourself. I talked with him about the enhancements I wanted to make and then provided him the code to merge into the app for the enhancements. He merged it into a test location and provided a link to me back the merged code for testing since I knew how I wanted it to work. After validation and finding and fixing bugs it was released.

I think sometimes when we make proposals or asks to a app we forget how much work may go into it. I have a feeling based on what i have looked into this isn't a minor ask and would require significant addition to the code and in turn significant effort.

3 Likes
17

:point_up:t2:

18

Perhaps an alternative, to avoid the extra work on the HE / HPM end could be to have a common approach to integrating library code into a dev's Git repository, giving them control to choose what libraries they want to use without impacting other repositories that want to use the same library. Not sure how this would work technically....

19

I think one approach is the developers could create their library bundle and then version it on the end of the name in Github. So for instance instead of Govee_Integration_V2.zip. Name it something like Govee_Integration_Library_V2.51.zip

Then in your Package manifest you simply include that bundle from the other repo. I don't think there is anything preventing that that from working. But this is thinking about folks using the library code above the developer making it for their own stuff.

The downside is now instead of just updating version tag in the manifest when i publish a new version i would now have to update the URL for it as well. Just a little bit more work. And if it is never used by others that is just a waste of my time. Small but still wasted. That said there are merits to the idea.

I also want to say that as @sburke781 stated It isn't that I don't see merits in the concept of what the OP is proposing but just have concerns about the ROI to implement it.

3 Likes
20

I have updated the proposal so that libraries would be versioned, and versions immutable. When you use a library you know it won't change, unless you as a developer update the the library version on the manifest file.

I'd be happy to develop it myself and create a PR, but I won't go through that effort knowing that the very idea wouldn't be accepted.

In the end, its like many tools, AI for instance, you can use it to create good things or bad things. But the tool is there.

To me, rejecting the idea would be like Hubitat folks thinking about community drivers and deciding not to make that possibility available because "developers could write bad code that would crash the hub".

1 Like