Beginner · Lesson 4 of 6 · 12 min
Hallucinations and how to verify output
Understand why models make things up, how grounding reduces it, and how to check output with effort that matches the risk.
Why models make things up
A model generates the most plausible continuation of your prompt. There is no built-in step that checks whether a statement is true. When it lacks the facts, plausible-sounding text fills the gap, and it sounds just as confident as a correct answer.
Hallucinations cluster in predictable places: specific numbers, dates, names, quotes, citations and links, anything after the knowledge cutoff, and niche details about your own business that the model never saw. Treat a confident tone as zero evidence.
Grounding: give it the source and limit it to the source
Grounding means putting trusted source material into the prompt and telling the model to answer only from it. A support assistant given your actual refund policy is far less likely to invent one than a model asked from memory.
Anthropic's guidance adds two simple instructions that make a large difference. First, explicitly allow the model to say it does not know. Second, restrict it to the provided documents rather than its general knowledge. Without permission to say 'not in the document', a model tends to produce an answer anyway.
At scale, retrieval-augmented generation (RAG) does the same thing automatically: it finds the relevant passages from your documents and grounds each answer in them.
<policy>
{{paste the refund policy here}}
</policy>
Answer the customer's question using only the policy above.
First, copy the exact sentence(s) from the policy that support your answer.
Then give the answer in plain language.
If the policy does not cover the question, reply exactly:
"The policy does not cover this. A team member will follow up."
Question: {{customer_question}}Make the output checkable
Ask for evidence alongside the answer. For long documents, Anthropic suggests having the model extract word-for-word quotes first and base its answer on those. For drafts, ask it to cite a supporting quote for each claim and to remove any claim it cannot support.
Another cheap check is to run the same prompt several times and compare. If the answers disagree on a fact, that fact needs checking. Consistent answers are not proof, but inconsistent ones are a warning.
Verify in proportion to the risk
Anthropic is clear that these techniques reduce hallucinations but do not eliminate them, so critical information still needs validating. Match the effort to what breaks if the output is wrong.
An internal meeting summary needs a skim. A customer-facing reply needs a person to read it before it sends. Anything with money, legal terms, health or contracts needs a person to check every number, name, date, quote and link against the source, and to open every URL.
Do maths in a spreadsheet or code, not in the model. If a workflow will make decisions without a person, start with full human review, measure how often the model is wrong, and only reduce review once you have the numbers.
Key points
- ◆Models have no truth check, so missing facts get filled with plausible text.
- ◆Hallucinations cluster around numbers, names, dates, citations, links and recent events.
- ◆Ground answers in supplied sources and explicitly allow 'I don't know'.
- ◆Ask for supporting quotes so output can be checked against the source.
- ◆Size verification to the risk and keep a person on anything customer-facing or high-stakes.
Terms in this lesson
Try it · exercise
Compare an ungrounded and a grounded answer on the same questions about a real document.
- Pick a one- or two-page policy or FAQ you can share safely, such as a public returns policy.
- Write five questions: three the document answers and two it does not.
- In a fresh chat, ask all five questions without pasting the document and save the answers.
- In another fresh chat, use the grounded template from this lesson with the document pasted in and ask the same five questions.
- Use your browser's find function to confirm each quoted sentence appears in the document.
You're done when: The grounded run returns the fixed 'policy does not cover this' reply for both unanswerable questions, and every quote in the other three answers is found word for word in the document.