Lesson 2 of 6 · 7 min read
How chatbots fail
Why a confident answer can be false, and which everyday tasks need a source you can open yourself.
Hallucinations
People say a chatbot “hallucinated” when it states something that is not true in a fluent, specific way. It may invent a book, a court case, a date, a quote, or a URL. The sentence can look exactly like a sentence that would be true.
This happens because the tool is built to continue text in a convincing way. It is not built to refuse every claim it cannot prove. Newer tools are better at saying “I don’t know.” They still miss.
The sure voice
The tone does not rise and fall with the evidence. A recipe and a tax rule can sound equally settled. If you ask it to be confident, it will often sound more sure without becoming more correct.
Ask the opposite instead: “If you are not sure, say so. Do not fill gaps.” Then check anything you would be ashamed to have gotten wrong.
Jobs that need a second look
Be slow to trust numbers, names, citations, laws, medical steps, and anything about a real person. The tool may mix two people who share a name, or describe a paper that was never written.
It can also be out of date. Training ended at some point, and even a live search can quote an old page. “Current” is a claim you can test by opening the source and reading the date.
What failure looks like in practice
A useful test: ask for the page, the statute, or the file the claim came from, and open it. If there is nothing to open, you do not have a source. You have a draft.
Another test: ask the same question twice with slightly different wording. If the dates or the steps change, you are watching the tool guess.
Keep these
- A hallucination is a fluent answer that is false, invented, or unsupported.
- Confidence is a style, not a measure of proof.
- Citations, math, laws, and facts about real people need a source you can open.