There was a interesting discussion at my job about AI and use cases. As you know product forums are plethora (Would you say I have a ‘plethora’ of piñatas?) of problems and solutions to products and why companies are not jumping on the AI bandwagon to leverage that huge knowledge base.
One of our data scientists started to model out a couple of forms that have large numbers of posts based on this suggestion. The bigger the data set the better right? Well yesterday afternoon he summarized his findings to the team. Here are some high level bullets:
Products being supported are all defective. Reason: ratio of forum posts is mostly issues and problems
Too many queries about issues receive answers from a account that doesn't demonstrate the knowledge to solve the issue. Reason: the larger the number of posts from an account the less likely posts will contain the solution.
Most issues are solved by power cycling or re-installing the product. Reason: many form posts end with power cycling or re-installed solved their problem.
As for discovery of real issues he did say there are some real gems and solutions but the create a Agentic AI sorting though all the noise in forums would be a full time job and because of the dynamics it would require some very complex prompting curbs.
It was very interesting to see all the raw numbers and relationships that were established with this non-work effort.
For the small amount of exposure I have had to the processes involved in honing a model, data scientists do still play a large part to refine it, identifying scenarios that are obvious to humans but not easily identifiable by machines (ATM). I would expect that over time, for use cases where it makes financial sense, there will be solutions to some of these more common situations.
Personally I would prefer to see humans identify the most useful information from the Community and see those nuggets make their way into the formal documentation, which could then have an AI model built from.
I think there is more than just information to be gained from an active participation in the Community. You get to form connections with other like-minded enthusiasts, learn those who you can trust to provide quality information and get a chance to contribute.
A forum is a community and filled with some very knowledgeable people in many cases. As you peel back the parameters you could eventually get to a usable source of trust within a community.
All in jest, a account I wouldn't trust is @rlithgow1 and we would have to write a rule that those responses are disruptive. While @gopher.ny would be a trusted account.
As I said it was a good experiment and there would be so much more if this would be something that could be brought to market easily but the complexity leaves these types of solutions to those companies that have the time and deep pockets to implement.
We started this journey months ago by experimenting with our own AI model, but it quickly became clear that the cost of maintaining it would far outweigh the benefits. So, we made the switch to an external platform — and the results have been incredible.
The new chatbot is able to transcribe our YouTube videos and deliver helpful answers directly to users. Even better, we’ve connected it to parts of the community and tapped into tens of thousands of resolved support inquiries. This means users get just the right amount of information to keep them moving forward.
Is it perfect? Not yet. But for anyone just starting their smart home journey with Hubitat Elevation, it’s an incredible valuable tool.
What this tells me is that we have reached a grange point in cost, effectiveness, and supportability. What I always find interesting is reading on successes and failures when implementing LLM's with RAG vs conversational with agentic capacities. It's baby steps vs. big bang and think in the long term that balance point will be achieved.
Glad to see this new feature. I'am hoping that this new AI chat offering will help those users with their basic questions that can be answered quickly by AI and offer weblinks to videos in some cases for better clarification.
But in this more advanced search example, the AI GPT just threw up it's digital hands and referred me to to manual means of disabling an app in the browser. In an identical search using the community forum search bar, I received the correct answer below. I am very grateful that we have very talented Hubitat users who respond and post correct responses and @bobbyD from HE support when we go down a rabbit hole...
"The more things change the more things are the same", IMHO.
I've been testing the chatbot against real-world support inquiries (non-community) and it has been 99% accurate in its responses. Sure, it has limitations with more advanced or nuanced issues, but in my experience, it immediately points users toward the specific documentation or video needed, saving considerable time and effort that was previously spent on manual searching.
came here looking for more info on the chat bot ~ didn't see any more recent threads so i figured this one was ok to comment on...
Anyway I've been trying the bot out recently - not for any serious needs, just curiosity to see how helpful it could be. I gather this versions is limited to knowledge based articles ala a fuzzy search versus some of the more "full blown" ai bots that are actually "learning" ... meaning an external bot returns way more info. I also seem to have stumped it into not responding at all ~ just the cycling dots or this:
It is allowed to access ChatGPT but, it is set to prioritize Hubitat sources, and is restricted to provide details on topics that could be confusing or otherwise misleading, though both could still occur as it is in continuous learning mode.
Well, I guess the bot has a slightly different opinion lol
. Nice.
or this:
