In 2023, two New York lawyers filed a legal brief citing six court cases as precedent. All six were completely fabricated – invented case names, invented judges, invented quotations, invented docket numbers. When the opposing counsel couldn’t locate any of them, a federal judge investigated, and it emerged that the brief had been researched using ChatGPT, which had confidently generated cases that never existed. When one of the lawyers later asked the same tool to verify its own citations, it confirmed they were real. That was also fabricated.
The lawyers in Mata v. Avianca were sanctioned $5,000 and required to personally notify every judge falsely named as the author of a fake opinion. The case became a founding reference point for AI use in legal practice, cited in nearly every AI-related sanctions ruling since. But the more useful lesson isn’t about legal ethics. It’s about why the underlying tool behaved this way at all, and why that behavior wasn’t a fluke.
What Is an AI Hallucination?
A hallucination is a confident, fluent factual claim generated by an AI model that has no basis in its training data or any verifiable source. It differs from a model simply being uncertain: a hallucinating model doesn’t hedge or flag doubt, because nothing in how it generates text distinguishes a confident guess from a verified fact.
The word “hallucination” is doing some misleading work here, since it implies a perceptual error, like the model is seeing something that isn’t there. What’s actually happening is closer to a well-calibrated guess landing on a wrong answer, dressed in the same fluent, assertive language the model uses for answers it happens to get right.
Why This Happens: The Statistical Explanation
In September 2025, OpenAI published a research paper titled Why Language Models Hallucinate, offering the clearest technical account yet of the mechanism. The paper’s core argument reframes hallucination not as a training failure to be patched out, but as a predictable statistical consequence of how these models are built and evaluated.
Two findings from the paper matter most for understanding the problem:
Generating an answer is a harder problem than checking one. A model might correctly recognize that a claim is false if you show it the claim directly, while still generating that same false claim unprompted when asked an open-ended question. The paper formalizes this gap mathematically: the rate of generation errors is bounded to be at least roughly twice the rate of classification errors on the same material.
Training rewards confident guessing over honest uncertainty. Most benchmarks and evaluation systems score a wrong, confident answer no worse than an “I don’t know,” and often score confident wrong answers higher than an honest refusal to guess. A model optimized against those benchmarks learns, in effect, that guessing is statistically the better strategy – even though a guess is exactly where hallucination comes from.
The paper draws a specific distinction that clarifies where hallucinations are most likely: patterns that repeat consistently in training data, like correct spelling or balanced parentheses, get learned reliably and rarely produce errors, because the model has seen the correct pattern often enough to internalize it. Arbitrary, low-frequency facts – a specific person’s birthday, a case citation, an obscure statistic – can’t be predicted from surrounding patterns the same way, and that’s where fabrication is most likely to fill the gap.
Why Better Training Doesn’t Fully Solve It
A natural response is to assume this gets fixed as models improve. OpenAI’s paper directly addresses this expectation and pushes back on it: accuracy will never reach 100 percent, because some real-world questions are inherently unanswerable from the training data available, regardless of model size or reasoning ability. A model that has never encountered a specific obscure fact has no way to derive it from first principles. It can either say so, or it can guess. Current training incentives, tuned against benchmarks that penalize hedging, tend to nudge it toward the guess.
This means hallucination isn’t a bug with a patch on the roadmap. It’s a structural property of a system trained to produce fluent, plausible-sounding continuations of text, evaluated by benchmarks that don’t reward saying “I don’t know.”
Does Retrieval-Augmented Generation Fix This?
Retrieval-augmented generation, commonly called RAG, is the most widely deployed mitigation: instead of relying purely on what the model memorized during training, the system retrieves relevant documents and gives the model that material as grounding before it answers.
RAG measurably reduces hallucination rates by anchoring the model to real source text rather than asking it to recall facts from memory alone. It does not eliminate the problem. A model using RAG can still misread the retrieved document, over-generalize from a partial match, blend two different sources incorrectly, or confidently state something the retrieved text doesn’t actually support. Grounding the model in real documents removes one failure mode without removing the underlying tendency to generate a fluent, confident answer regardless of certainty.
A Practical Framework for Spotting Hallucination Risk
Not every task carries equal hallucination risk. A rough way to think about it:
| Type of claim | Hallucination risk | Why |
|---|---|---|
| Common, well-documented facts (major historical dates, basic definitions) | Low | Appears consistently across training data |
| Summarizing a document you provided | Low to moderate | Depends on the document’s length and how information is distributed within it |
| Specific citations, case law, academic references | High | Exact names, numbers, and sources are exactly the low-frequency detail the OpenAI paper identifies as hardest to predict reliably |
| Niche statistics or obscure biographical facts | High | Rarely repeated enough in training data to be reliably memorized |
| Anything the model is asked to verify about its own prior answer | High | The model checking its own work has no independent source to check against |
How to Reduce Your Own Exposure to This
- Treat specific citations, numbers, and names as claims to verify, not facts to trust, especially in legal, medical, or financial contexts where a fabricated detail carries real consequences.
- Never ask the same model to verify its own previous claim as your only check – this is precisely what failed in Mata v. Avianca, since a model with no grounding in reality can confidently confirm its own fabrication.
- Ask for sources, then check that the sources actually exist independently, rather than trusting a citation format that merely looks correct.
- Be more skeptical of confident answers to obscure questions than confident answers to common ones, since the underlying mechanism makes rare facts the highest-risk category.
- Use tools with retrieval or citation features for anything fact-sensitive, understanding that this reduces but doesn’t eliminate the risk.
Frequently Asked Questions
Is hallucination the same as the model lying? No. Lying implies the model knows the truth and states something else deliberately. A hallucination is a confident guess presented with the same fluency as a correct answer, with nothing in the generation process to distinguish the two internally.
Will hallucinations eventually be solved completely? Current research, including OpenAI’s own paper on the subject, argues this isn’t realistic. Some questions are unanswerable from available training data regardless of model size, so a 100 percent accuracy rate isn’t achievable even in principle.
Do bigger, newer models hallucinate less? Generally yes on common, well-represented facts, since larger models and more training data improve reliability on patterns that repeat often. Rare, specific, or highly technical facts remain a weak point regardless of model size, for the reasons described above.
Does asking the model to “double check itself” help? It can catch some inconsistencies, but it isn’t reliable as a sole safeguard, since a model with no external source to check against can hallucinate a confident confirmation of its own error, which is exactly what happened in the Mata v. Avianca case.
Are some topics more prone to hallucination than others? Yes. Specific citations, statistics, names, and dates are consistently higher-risk than general explanations or well-documented common knowledge, because the underlying mechanism struggles most with low-frequency, hard-to-predict details.
The Practical Takeaway
Hallucination isn’t a defect that disappears with the next model release. It’s a predictable outcome of training a system to generate fluent, plausible text and then evaluating it in a way that rewards confident guessing over honest uncertainty. The practical response isn’t to distrust every AI-generated claim equally – it’s to know which categories of claim are highest-risk, and to verify those independently before they matter. Our own approach to separating verified fact from AI-assisted output is set out in full in our fact-checking policy and AI editorial policy. For a related failure mode worth understanding alongside this one, see our explainer on context windows and why models can lose track of information even when it’s technically available to them.