AI models make things up because they are built to produce plausible text, not to check facts, and because training and testing have tended to reward confident guessing over admitting uncertainty. You catch the errors by never publishing unchecked claims: ground the model in documents you supply, ask it to cite, and have a person verify every name, number, quote and link before anything goes out.
Why it happens
Researchers argue the problem is partly built into how models are trained and scored. An OpenAI-authored paper on arXiv says language models hallucinate because training and evaluation reward guessing rather than admitting uncertainty, and that errors arise naturally when a model cannot reliably tell incorrect statements from facts. The authors describe models as optimised to be good test-takers and propose changing how benchmarks score uncertain answers.
For a business user the practical lesson is simple: fluent, confident wording tells you nothing about accuracy.
Common failure modes in business work
- Invented facts and figures: statistics, dates, prices or product specifications that sound right but have no source.
- Fabricated citations: references with real-looking titles, authors or web addresses that do not exist or do not say what is claimed.
- Misattributed quotes: words credited to a real person who never said them.
- Outdated information: a model may reflect old rules, prices or product versions.
- Plausible but wrong summaries: a summary that adds a point the document never made, or drops a caveat.
- Overconfident legal or regulatory statements: rules described as settled that vary by jurisdiction or have changed.
- Inconsistency: the same question gives different answers on different runs.
Reduce errors before they happen
Anthropic’s guide on reducing hallucinations lists several techniques that apply broadly to prompting any model:
- Allow it to say it does not know. Give explicit permission to admit uncertainty.
- Ground the answer in quotes. For long documents, ask for word-for-word extracts first, then base the analysis on them.
- Require citations and retraction. Ask the model to find a supporting quote for each claim and to remove any claim it cannot support.
- Restrict the knowledge source. Instruct it to use only the documents provided, not its general knowledge.
- Compare multiple runs. Inconsistencies across repeated outputs can indicate hallucinations.
The same page is candid that these techniques significantly reduce hallucinations but do not eliminate them, and advises always validating critical information, especially for high-stakes decisions. Treat these as risk reduction, not a guarantee.
A verification routine you can adopt
- Classify the content by risk. A social caption is low risk. Pricing, legal, medical, financial or safety content is high risk and needs expert review.
- Highlight every checkable claim. Names, numbers, dates, quotes, product specifications, legal statements and links.
- Verify against primary sources. Open the actual source: the regulator, the manufacturer, the original document. Do not ask the same model to confirm itself.
- Check every link and citation. Open each one. Confirm the page exists and states what the text says.
- Remove what you cannot verify. If you cannot find support, delete the claim or label it clearly as unverified.
- Record the sign-off. Note who checked what and when. See human-in-the-loop workflows for how to assign this responsibility.
- Spot-check regularly. Sample published work after the fact to see whether errors are slipping through.
Governance, not just habits
Individual care helps, but organisations need policy. The US National Institute of Standards and Technology’s AI Risk Management Framework page notes a Generative AI Profile (NIST-AI-600-1, released July 2024) intended to help organisations identify unique risks posed by generative AI and propose risk management actions. The page does not itself detail specific risks, so read the full profile if you are building a formal approach.
Sensible internal rules include: a list of approved tools, a ban on pasting confidential data into public tools, mandatory human review before publication, and an owner for AI-related incidents.
The hidden cost of checking
Verification takes time, and that time belongs in your estimates. Teams that skip it save minutes now and pay later in corrections and lost trust. Our article on hidden costs of AI content production explains how to budget for review and rework, and AI ethics and brand trust covers why customers notice mistakes.
Practical limits
- A fact-check cannot catch what you do not examine. Read the whole output, not only the numbers.
- Retrieval from your own documents lowers risk but the model may still misread them.
- For regulated advice, AI output is a draft for a qualified person, never the final word.
Where Radiance Impact fits
Radiance Impact’s Technology division builds workflow automation and AI-assisted tools with human review steps built in. See Technology or the automation demos, or contact us to discuss a review workflow for your content.
Sources
- Why Language Models Hallucinate (arXiv abstract), arXiv
- Reduce hallucinations, Anthropic documentation
- AI Risk Management Framework, NIST
