AI Hallucinations: Why Models Confabulate

LLMs don't have intent. But they can confabulate.

The difference matters. Lying requires intent. Confabulation is gap-filling: constructing plausible-sounding narratives to bridge missing knowledge. Humans do this too (ask any eyewitness).

Some researchers argue that "hallucination" is an imprecise label - and that a lot of what we call hallucinations are better understood as confabulations: coherent narratives that happen to be false. An ACL 2024 paper explores this framing in depth.

---

Why This Happens

It comes down to how LLMs work: next-token prediction.

The model doesn't ask "is this true?" It asks "what token is statistically likely to come next?" If the most probable next token leads somewhere false, the model follows that path confidently.

Key mechanisms:

1. Statistical plausibility over truth - the model optimizes for "sounds right," not "is right"
2. Snowball effect - one wrong token early can cascade into a coherent-but-false narrative
3. Training incentives - evaluation setups often reward guessing over saying "I don't know"
4. No verification mechanism - there's no internal fact-checker

---

What I've Noticed

Hallucinations are most likely when:

• You ask about obscure topics - less training data means more gap-filling
• You request specific details - dates, numbers, citations are high-risk
• The model "should" know but doesn't - it fills gaps rather than admitting uncertainty
• Context is ambiguous - the model picks a plausible interpretation
• You push for an answer - "give me something" triggers confabulation

---

Common Patterns

The Fake Citation

"According to Smith et al. (2023) in the Journal of..."

The paper doesn't exist. The journal might not even cover that topic. The model generated a plausible-sounding reference.

The Confident Wrong Answer

"The population of X city is 2.4 million."

Stated with zero hedging. Often close enough to seem believable.

The Plausible Fabrication

"The company was founded in 1987 by..."

The narrative sounds reasonable. The facts are invented.

---

How to Reduce Hallucination Risk

In your prompting:

1. Ask the model to cite sources - then verify them independently
2. Request uncertainty markers - "say 'I'm not sure' if you don't know"
3. Break complex queries into verifiable steps
4. Avoid asking for specific numbers/dates unless you can cross-check

In your workflow:

1. Treat LLM output as a first draft, not ground truth
2. Verify factual claims against authoritative sources
3. Be especially skeptical of citations, statistics, and historical details
4. Use retrieval-augmented generation (RAG) for fact-dependent tasks

---

Why This Matters

Hallucination isn't something you can eliminate entirely. It's tied to how next-token prediction works.

Models that generate coherent text will sometimes generate false text. The same mechanism that enables fluent writing enables confident confabulation.

Engineer takeaway: Don't ask "will this model hallucinate?" Ask "how do I catch it when it does?"

---

My Take

I've learned to think of LLM outputs like a very eager research assistant who often doesn't admit uncertainty. They'll give you something - but you need to verify anything that matters.

The goal isn't to eliminate confabulation. It's to build workflows that detect and correct it before it reaches production.

---

Further Reading

• Sui, P., Duede, E., Wu, S., & So, R. (2024). Confabulation: The Surprising Value of Large Language Model Hallucinations. ACL 2024.
• Farquhar, S. et al. (2024). Detecting hallucinations in large language models using semantic entropy. Nature.
• Kalai, A. T. et al. (2025). Why language models hallucinate. OpenAI.