Searching for suggestions and feedback; enabling wife to create automations via AI (LLM)

💬 Lounge
1

Hello, after a long time I finally managed to sit and create a Proof of concept of my project. Here goes an explanation on what it does, I'd really appreciate suggestions of use cases or anything related to you might consider useful.

Problem Statement: It is only me who creates automations in my smart home. I want to enable the rest of the family to easily interact with it, creating ephemeral (single use) or recurring automations. I also believe that Smart homes can be built in a way that foster healthier life-style by helping us doing the right thing (taking the trash out, go to bed on time, water the plants, etc).

My solution: I created a webservice integrated with an AI agent that operates the smart home via Maker API. In practice this means a chat webpage where you can interact with the smart home, do commands and create automations, via text.
Behind the curtains, smart home operations (turn off lights, etc) are done immediately via MakerAPI, while automations are stored (as a JSON object with certain fields and values) and a webservice that listens to my smart home events monitors when is it time to trigger the automations.

Note that these automations are not interpreted or stored by Hubitat. I can discuss more about its characteristics later if necessary but that's besides the point now.

My ask: do you perceive value in such a solution that would enable your family to interact with the smart home like this? What other integration do you think would be useful for the agent to have and why? (e.g. Google Calendar, Alexa Shopping List, Todoist, etc).

If you are really interested and would like to give it a try, let me know and I create an account for you, but keep in mind this is not a commercial product.

Thanks!

1 Like
2

If you just want an answer either way, this not something I would be interested in, and my family doesn't really have your use case.

Really though, Alexa+ does much of this already. Using the devices I share with the echo skill, I can tell Alexa to create a routine to do simple things with devices, based on motion, or on a schedule, or just today, which seems very similar to what you are proposing.

Is your integrated AI agent an LLM that runs locally on your server? Ultimately what I would want is to run a small LLM locally in my house, like we try to do with everything else on Hubitat. This seems totally possible now. Seems like a local LLM could just be put in learning mode for a bit, and learn our routines while we do things manually for a week, or watch what our current automations do and learn from that.

If the solution is also a business proposal, I don't think there is much to be made from this, since it is already there with major platforms like Amazon and Google. Running your own LLM servers might be the way to go, if your idea is to invest in small server farm to offer AI smart home services directly, for people who don't want to be on a major platform's AI.

3

Tks for sharing it. I'll try to cover your thoughts in a few bullet points.
-To be fair, I had this project idea even before Alexa+ was launched. Even to this day it is still not available in my country and quite frankly, I don't feel like paying for it. I use a Alexa a lot, but I don't like how they change functionalities arbitrarily (for example removed the API for Alexa's lists). That said, I know that this project will not replace Alexa, maybe a more professional version of this project combined with Home Assistant Voice product would.
-Still about Alexa, there are lots of devices that it is just unable to interact with, at least this project allows you to interact with any property of any device that exists in Hubitat.

-Right now I use any commercial LLM (OpenAI, Claude, etc). In theory it could easily be switched to a smaller LLM and run locally, but I haven't tried that (the main challenge is making sure that a potentially smaller context window would not harm much of the experience). I do this project more as way to help me start sharp on the latest techniques for AI Agents, right now I don't intend deploying local LLMs.

-I saw in another thread a person mentioning something that could be a valid use case for me: using the LLM to help identify and troubleshooting faulty devices. Dunno how the troubleshooting would work but the part of identifying potentially faulty (non responding) devices should be quite feasible.