I have used it for editing white papers and initially the hallucinations were horrendous. I now add, "include hyperlinks to references for any assertions made" in every prompt I use for reviewing. This has cut down drastically on it even making wrong assertions. Since I know the subject matter pretty well, I can spot when it links to a bad source pretty quickly now. So, when it does hallucinate, I see it almost immediately.
And this is where the issue is. If you are using it to bridge a knowledge gap, it just isn't there and you are stepping into dangerous territory if you rely on it. If you already know the material, and just want to make quicker work of a task, it is great as long as you know enough to spot the errors.
I agree. AI is a tool, even though its capacity for generating written (or spoken) words makes it act in some ways like a skilled and/or knowledgeable person.
If the (actual) people using a tool are themselves unskilled and/or lack relevant knowledge, then the tool is more likely to be used ineffectively or even dangerously.
Some of those people using the tool won’t even realize it’s being (mis)used that way, and that to me is the scariest part of this brave new world.
The code it gives almost never works right the first time, but when it does, it still does something in some way I didn't want it to, and I have to tell it specifically what I wanted.
For Hubitat Groovy code, I already was coding my own stuff for a couple years before AI advanced to where it is now, so I now pretty much just tell it exactly how I want it to write it, including names I want to use for attribute names, method names, if it needs to add a refresh button or interval polling, if it should create child devices or put it all in one driver. I also tell it to add lots of debug statements for testing, and also what I want to see as options for changing things in preferences.
It takes some thought to make a prompt that will get you what you want without needing too many iterations of it having it fix the code. Still, I think my record is three iterations to get a driver working how I want it to.
You can include, in the AI prompt, words telling it to tell you it doesn't know...
...just as described above related to made up references.
If you're asking AI to create something and it doesn't have a 300 word Prompt, you're going to spend a lot of time debugging. So adding another sentence to limit it's flights of fancy, will prevent at least some of the hallucinations.
I went out on a limb recently and asked Claude to write a groovy app for me since the logic I had in mind would have been difficult or impossible for me to put together myself in rule machine; I could have asked Claude to design a rule for me but then I'd still have to create it manually in the hub UI and that would be tedious.
Initially, the notes were vague about authorship, with Claude writing only that the code was "written for Mark." I instructed it to "make it clearer in the notes at the start of the file that Claude wrote this for Mark" and here's what it added:
* ---------------------------------------------------------------------------------------------------
* AUTHORSHIP
* This app was written by Claude (Anthropic) for Mark, in a conversation on 2026-08-31. Mark specified
* the behavior; Claude designed and wrote the code. It has NOT been reviewed line-by-line by a human,
* and had not been run on a hub at the time it was written — read it before you install it, and watch
* the logs through the first few cycles.
* ---------------------------------------------------------------------------------------------------
It'd be nice if anyone that uses an LLM to create code that they themselves can't understand and vet (like me) added similar language before sharing it here. Probably not likely to happen consistently, though.
In addition to these great suggestions, I also leverage a different AI model to validate. Have one AI model generate the code, then provide that code to a different AI model ... without any context, have it analyze and describe it.