I just ordered some HA Voice PE / ESP32-S3 Dev Kits. Planning to try and get local voice control via OpenClaw, Ollama, Wyoming Satellite, etc. Since I know nothing about this I'm sure it's going to be fun. I will keep you updated.
@simon4 ?
I hope you are going to include some Red Dwarf references.... 
EDIT - I now, well after I posted this, that I got my cultural reference wrong... 
Still, any Red Swarf references are still welcome.
1 Like
I am in the throws of implementing this and it has been a wild ride so far. I am using OpenClaw (named Jeeves) to act as the glue connecting everything. I have that installed in a docker container on a private network and I control the access so it can't do all of the things that people are freaking out about.
I tried connecting Jeeves to multiple local models but it never really worked and/or response was too long so I ended up using Claude. I intend to come back to that at some point. I have an Nvidia RTX 2000 Ada so I am not lacking in GPU power.
I also have two-way setup with Jeeves through a Telegram Bot which works great and allows me to test everything without having to go through the convoluted pipeline discussed below.
I developed a set of tools for it that run on another docker container that allow Jeeves to access all Hubitat endpoints and a browserless tool that allows it to access the Hubitat interface to get the many things that Hubitat doesn't allow through endpoints. I also extract all information from my four hubs each night and store them in a Postgres db to speed things up. So Jeeves can quickly find device statuses, change settings, find rules and trace them, etc.
Now I am adding voice and have Jeeves talking to Home Assistant right now. I created four Docker containers on Unraid as the voice stack. Home Assistant Container acts as the orchestrator and runs a custom integration that forwards conversation text to the OpenClaw container over loopback. Faster-Whisper provides GPU-accelerated speech-to-text. Piper generates spoken audio responses using a British male voice. OpenWakeWord listens for a wake word to trigger the pipeline. I do not have the Voice PE working with this - it is next on my list.
I went the unraid approach as my main Home Assistant implementation, which does very little in my setup other than connect to devices I can't bring directly into Hubitat, runs on a Green and everything I read said it was too slow to do anything with voice.
I am also actively looking for better devices to take voice commands and circling the project that takes old Google NESTs and replaces the board but uses their mic and speaker array.
I can't say whether my approach is the most efficient way to do this but it is working. Hope that is helpful?
7 Likes
Your posts are a large part of my inspiration to give local voice control a try. You are clearly vastly more knowledgeable about this than I. If you are struggling It may be impossible for me. I decided to undertake this project at 3 am in the morning - which is always a red flag!
I will post my progress - if any.
Thanks!
3 Likes
Just keep posting progress notes here about planning on getting started...that's what I'd do.
3 Likes
USPS texted AmeriDroid shipped on 2/23.
AmeriDroid responded promptly with an update on arrival time.
This is a good omen. 
3 Likes
HA Voice PE / ESP32-S3 Dev Kits arrived yesterday! Smaller than expected - story of my life. Planning to convert my old Windows 10 Intel i9 9900K 32 Gb PC to Ubuntu 24.04 LTS for starters.
I want to get simple Voice Control to Home Assistant in a Docker container on the 9900K to Hubitat C8 to Hubitat devices working first and then integrate OpenClaw after that.
Grok is helping.
Comments, suggestions, and snarky remarks welcome.
I'm going in!
2 Likes
Really looking forward to getting started!!
1 Like
What have you gotten going so far?
My current state - Jeeves running across my entire HA setup. Access through Home Assistant Voice but haven't figured out how to change the wake word yet so stuck with Hey Jarvis. Jeeves is a great notification engine as can monitor everything and make decisions based on what he is seeing. Can directly create automations in Home Assistant although that is so far not the greatest but still playing with that. I have sniffed out every endpoint I can find on Hubitat to give him the ability to do a lot there. I also built a browser less container that he can use to get to other things. Wish there was an API on hubitat to allow for editing of rules as right now he has to tell me what to do vs doing it himself. Open-claw is still buggy - he often forgets things. The last week he has been constantly getting dates wrong. and combining the wrong day with date.
FWIW, my small group has noted that the LLMs chatgpt & gemini do not do well with dates --and not just sorting. Given a list of medical record events and their dates, gemini re-assigned some of the dates to the wrong event.
That's referred to as a hallucination. It's an AI thang.
Yesterday I got my PC up and running Ubuntu. I gathered all of the best parts from my computer junkyard to repurpose my old Intel i9 9900K box for this project. Using a 1Tb Samsung 9100 Pro NVME for the boot drive and added a 12Tb Ironwolf for storage.
Setting up Docker and installing component programs today. As I said, Grok is helping.
3 Likes
Working!
Echo / Alexa still working as well (WAF) but now I can talk to my little white box - "OK NABU - Turn on the bedroom light." - and the light turns on!
The thrill is almost like my first day with SmartThings - only extremely more difficult to set up.
I'm screaming at Vicki - IT'S ALL LOCAL! NO INTERNET. THIS IS ALL COMING FROM A BOX IN THE OFFICE! (home office). She is looking at me - unimpressed/skeptical/I've lost my mind. - Possibly ...
4 Likes
According to YouTube videos and other reliable sources you just open up a terminal in Ubuntu and run a few "copy and paste" command lines and - everything just works!
That was not my experience.
Admittedly, what little I know about Linux had faded and I am not an expert at any part of this process but it was a real struggle. I could not have done it without Grok. Many of my struggles were related to just setting up Ubuntu - simple steps like installing Git and other basics. Then, I have only used Docker for a few hours attempting to set up something on a Raspberry Pi. I had all kinds of sticky issues with the various components accessing one another's Docker containers, etc. It's finally working. Docker / Home Assistant / Wyoming / Faster Whisper / Piper / Ollama -I can run LLM models up to about 7 or 8 Gb with no problems or (much) lag. Did you know that LLM stands for Long Lag Models?
So, working on OpenClaw now.
I know - I'm worried about Alexa and privacy and then I give a strange man I found wandering around the internet - and his bag of different colored pills - a permanent home at the controls inside my computer. What could go wrong?
I will keep you updated!
2 Likes
Good progress. I now have three bots running in openclaw. Jeeves who manages all things house. Wooster, my project manager, and Thor who does all of the coding and testing.
I recently set Thor a task to find and develop tools for every endpoint in Hubitat. This coupled with a headless chrome container and CDP running on another machine give Jeeves the ability to do most things in Hubitat. But it’s quirky and I am beginning to look into node red to allow Jeeves full autonomous control.
Voice is a challenge. It’s all connected through home assistant but slow. I am working on the wake word speed and playing with the idea of having Jeeves hit a faster local model to service requests and hand them off to another bot for deep dive. So you can automate things like lights on and off etc quickly. Also there are several voice hardware platforms out there developing but none ready for prime time that I have found.
3 Likes
I can turn on the bedroom light!
2 Likes
Yes. There seemed to be some easy ways to do that - like having two VA devices with different wake words connected to different LLMs. Not elegant and not cost-effective. I will be following to see how you manage to implement an elegant solution!
2 Likes
Using the system for local AI chat is really slowed down with HA overhead added - even if I'm not asking for any smart home tasks. Just running Ollama / Montana / Piper / Whisper was much faster and chatting was fun and sometimes amazing. I'm just using an old NVIDIA 2700 GPU with 8 Mb of VRAM. I'm not going to spend $4K or more on better hardware.
So, figuring out a way to run smart home tasks with a smaller LLM while switching to a larger model for purely AI fun with no HA overhead is surprisingly desirable. I did go back and uncheck many boxes in the Hubitat / Home Assistant integration app.
The voice recognition is not comparable to the Echo devices. Device names are frequently misunderstood. The quality of the sound output is very close to bad on the tiny speaker. There are a number of speaker upgrade projects around but that is not at the top of the list yet. I expect that the sound portion of the little Preview Edition is not audiophile quality anyway.
My integration of OpenClaw is still crawling along. I feel like I'm exploring a volcano close-up with that process.
I do like to hear about your progress even if you are just a little speck on the horizon ahead of me!
2 Likes
Still having fun!
This is addicting. Added OpenWakeWord so I can use my own wakewords instead of "OK Nobu". Sort of working but recognition is not as good as the wakewords built into the Voice Assistant ESP32. I'm not ready to experiment with custom flashing of the device firmware.
Also having fun trying out different LLM's with Ollama. I am constrained by the 8 Gb VRAM in the NVIDIA RTX 2700.
I am struggling to get OpenClaw implemented. I keep running into roadblocks related to the Docker containers.
Connection to the Hubitat C8 works very well. I don't think that the Preview Edition hardware will ever compete successfully with the Echo devices - unless I lose the internet connection!
So, progress!
1 Like