LLM · RELIABILITY
What Is an AI Hallucination? Why AI Makes Things Up
An AI hallucination is a statement from a language model that reads as fluent and sure of itself but is false or unsupported. It happens because the model writes the most plausible continuation of the text, and nothing in that step checks whether the result is true. Grounding and verification reduce it. Nothing removes it.
What counts as a hallucination
A hallucination is a plausible falsehood: text that sounds right and is wrong, or that claims something no source supports. The word comes from research on language generation, where surveys use it for generated text that is unintended or unfaithful to its source.
A documented case: in 2023 two New York lawyers filed a court brief that cited judicial opinions produced by ChatGPT. The court found the opinions did not exist, quotes and citations included. When one lawyer asked the chatbot whether a cited case was real, its reply said it had supplied real authorities that could be found in standard legal databases. The reply was more generated text, and it was wrong.
Researchers do not all use the same word. Kalai and colleagues use hallucination for plausible falsehoods in general. A 2024 paper in Nature narrows it: it calls a model's claims confabulations when they are both wrong and arbitrary, meaning the wrong answer shifts with irrelevant details such as the random seed, and it argues that putting every error under one label is unhelpful. Both words point at the same visible behavior, a fluent answer that is not true. This page uses hallucination because it is the term most people search for.
Why it happens: plausible is not the same as true
A language model writes one token at a time. At each step it scores every possible next token, and one is picked. The score measures how well a token fits the text so far, given what the model absorbed in training. How that works is covered in how next-token prediction works.
That is the whole job of the step. It contains no lookup into a database of facts and no comparison with the real world. When the training text made the correct continuation the likeliest one, the answer comes out right. When it did not, the model still produces a likely-sounding continuation, in the same fluent style.
Rare facts are the weak spot. A study of question answering found that a model's accuracy on a fact tracks how many documents about it appeared in the training data (Kandpal et al., 2023). Kalai and colleagues make a related argument for facts with no pattern behind them, such as a person's birthday: the share of facts seen exactly once in training sets a floor on how often a base model gets that kind of fact wrong.
Why models answer instead of saying "I don't know"
After pretraining, models are tuned and then compared on benchmarks. Kalai and colleagues argue that most of these tests give one point for a right answer and zero for a wrong answer or for "I don't know". Under that grading, a guess can only help. It has some chance of scoring, and abstaining scores nothing. A model tuned and ranked this way learns to answer even when unsure.
Their example: asked for one author's birthday and told to answer only if it knew, a model gave three different dates across attempts, all wrong. Their proposed fix is to change the scoring so a confident wrong answer costs more than saying "I don't know". This is one research team's argument, not a settled result.
Where temperature fits
Temperature changes how often a lower-ranked token gets picked, covered in how next-token prediction works. It is not the cause of hallucination, and turning it down is not the cure. At the lowest setting the model keeps taking its top-scoring choice, and if that choice leads to a wrong statement, the statement is wrong every time.
Temperature changes how varied the answers are. It does not change what the model learned. Variation is still useful as a warning sign: when the same question gets different answers across repeated tries, the model is probably guessing, which is the idea behind the semantic entropy method in the Nature paper.
What reduces it, and what does not
Grounding gives the model text to work from. Retrieval-augmented generation fetches relevant passages and puts them in the prompt, so the answer is written from supplied text instead of memory. In the original RAG paper, retrieval models produced more factual language than a model without retrieval on generation tasks (Lewis et al., 2020). The what is RAG page covers the mechanism.
It reduces the problem. It does not remove it. A 2024 preregistered study of commercial legal research tools built on retrieval found that they still produced hallucinations, and that the vendors' claims of hallucination-free results were overstated.
Asking for sources helps only if you open them. A citation has the right shape whether or not the document exists, and a model's confirmation that a source is real is also generated text. The check has to come from outside the model: open the document, search the exact claim, or compare the answer with the source.
No method here guarantees a correct answer. Treat any claim that matters as unchecked until it has been checked against a source.
FAQ
- Why does AI make things up?
- A language model writes the most plausible continuation of the text so far. Nothing in that step checks the result against the real world, so a fluent but false statement is produced the same way as a true one. Rare facts and answers the model was never rewarded for declining make it more likely.
- Is an AI lying when it hallucinates?
- Lying means saying something in order to mislead. The mechanism here is text generation, and nothing in it aims at misleading anyone. The output can still mislead you, which is why claims that matter need checking.
- Can AI hallucinations be eliminated?
- Current methods reduce them. Grounding in retrieved text helps, and so do changes to how models are scored. The studies cited on this page found that tools built on retrieval still produced hallucinations, so any claim of zero is unproven.
- Can I trust what a chatbot tells me?
- Treat it as a draft. Names, numbers, dates, quotes and citations are the specific, rarely seen facts where errors concentrate, so check those against a source before you rely on them.
- Are fake citations a kind of hallucination?
- Yes, and a common one. A citation has a recognizable shape of authors, title, venue and year, and a model can produce that shape for a paper that does not exist. The 2023 Mata v. Avianca court record is a documented example. Open the link or search the exact title before you cite it.
Sources
- Kalai, Nachum, Vempala and Zhang, "Why Language Models Hallucinate" (2025)
- Ji et al., "Survey of Hallucination in Natural Language Generation", ACM Computing Surveys
- Farquhar et al., "Detecting hallucinations in large language models using semantic entropy", Nature (2024)
- Kandpal et al., "Large Language Models Struggle to Learn Long-Tail Knowledge" (ICML 2023)
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020)
- Magesh et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools" (2024)
- Mata v. Avianca, Opinion and Order on Sanctions, S.D.N.Y., 22-cv-1461 (June 22, 2023)
Related
Last updated 2026-10-11