I just ordered some HA Voice PE / ESP32-S3 Dev Kits. Planning to try and get local voice control via OpenClaw, Ollama, Wyoming Satellite, etc. Since I know nothing about this I'm sure it's going to be fun. I will keep you updated.
@simon4 ?
5 Likes
I hope you are going to include some Red Dwarf references.... 
EDIT - I now, well after I posted this, that I got my cultural reference wrong... 
Still, any Red Swarf references are still welcome.
1 Like
I am in the throws of implementing this and it has been a wild ride so far. I am using OpenClaw (named Jeeves) to act as the glue connecting everything. I have that installed in a docker container on a private network and I control the access so it can't do all of the things that people are freaking out about.
I tried connecting Jeeves to multiple local models but it never really worked and/or response was too long so I ended up using Claude. I intend to come back to that at some point. I have an Nvidia RTX 2000 Ada so I am not lacking in GPU power.
I also have two-way setup with Jeeves through a Telegram Bot which works great and allows me to test everything without having to go through the convoluted pipeline discussed below.
I developed a set of tools for it that run on another docker container that allow Jeeves to access all Hubitat endpoints and a browserless tool that allows it to access the Hubitat interface to get the many things that Hubitat doesn't allow through endpoints. I also extract all information from my four hubs each night and store them in a Postgres db to speed things up. So Jeeves can quickly find device statuses, change settings, find rules and trace them, etc.
Now I am adding voice and have Jeeves talking to Home Assistant right now. I created four Docker containers on Unraid as the voice stack. Home Assistant Container acts as the orchestrator and runs a custom integration that forwards conversation text to the OpenClaw container over loopback. Faster-Whisper provides GPU-accelerated speech-to-text. Piper generates spoken audio responses using a British male voice. OpenWakeWord listens for a wake word to trigger the pipeline. I do not have the Voice PE working with this - it is next on my list.
I went the unraid approach as my main Home Assistant implementation, which does very little in my setup other than connect to devices I can't bring directly into Hubitat, runs on a Green and everything I read said it was too slow to do anything with voice.
I am also actively looking for better devices to take voice commands and circling the project that takes old Google NESTs and replaces the board but uses their mic and speaker array.
I can't say whether my approach is the most efficient way to do this but it is working. Hope that is helpful?
7 Likes
Your posts are a large part of my inspiration to give local voice control a try. You are clearly vastly more knowledgeable about this than I. If you are struggling It may be impossible for me. I decided to undertake this project at 3 am in the morning - which is always a red flag!
I will post my progress - if any.
Thanks!
3 Likes
Just keep posting progress notes here about planning on getting started...that's what I'd do.
3 Likes
USPS texted AmeriDroid shipped on 2/23.
AmeriDroid responded promptly with an update on arrival time.
This is a good omen. 
3 Likes
HA Voice PE / ESP32-S3 Dev Kits arrived yesterday! Smaller than expected - story of my life. Planning to convert my old Windows 10 Intel i9 9900K 32 Gb PC to Ubuntu 24.04 LTS for starters.
I want to get simple Voice Control to Home Assistant in a Docker container on the 9900K to Hubitat C8 to Hubitat devices working first and then integrate OpenClaw after that.
Grok is helping.
Comments, suggestions, and snarky remarks welcome.
I'm going in!
2 Likes
Really looking forward to getting started!!
1 Like
What have you gotten going so far?
My current state - Jeeves running across my entire HA setup. Access through Home Assistant Voice but haven't figured out how to change the wake word yet so stuck with Hey Jarvis. Jeeves is a great notification engine as can monitor everything and make decisions based on what he is seeing. Can directly create automations in Home Assistant although that is so far not the greatest but still playing with that. I have sniffed out every endpoint I can find on Hubitat to give him the ability to do a lot there. I also built a browser less container that he can use to get to other things. Wish there was an API on hubitat to allow for editing of rules as right now he has to tell me what to do vs doing it himself. Open-claw is still buggy - he often forgets things. The last week he has been constantly getting dates wrong. and combining the wrong day with date.
FWIW, my small group has noted that the LLMs chatgpt & gemini do not do well with dates --and not just sorting. Given a list of medical record events and their dates, gemini re-assigned some of the dates to the wrong event.
That's referred to as a hallucination. It's an AI thang.
