Hi there! I built something siimlar as an integrated feature of my HousePanel dashboard application. After building it I had the idea of generalizing it and making an app separate from my dashboard so I started with a search to see if anyone had done this already, and that led me to you. So wow - great minds think alike.
My current implementation is intricately linked in with my dashboard app and on the back end calls the Open AI api. It uses the Responses api and sends context based on prior smart home interactions as captured by subscriptions. I also wrote a function to query all devices on the hub and tied that into the Responses api and it works great. I assume you are doing something similar.
I look forward to checking out what you've done. Let's compare notes so we can learn from each other as both of our projects evolve. You can read about my work here: