Hallucination

A model cannot internally distinguish a recalled fact from a plausible invention, so confident-sounding fabrication is a structural property of generation, not an occasional bug.

An LLM is a document-completion engine that mimics its training data: given a prompt, it emits the statistically most likely continuation, with no internal flag distinguishing "this is a recalled fact" from "this is a plausible-sounding invention." A fabricated detail — a URL, a citation, a person's name — is emulated with the same confidence as a true one, because emulating exactly this kind of plausible-looking content is what training optimizes for. Directives like "don't make stuff up" have limited effect precisely because the model has no internal signal to condition on: from its own perspective, a hallucination and a correct answer are generated by the identical process.

Two complementary causal accounts for why it happens, from the sampling and generation process itself rather than a training-data gap alone:

  • Self-delusion — the model cannot distinguish prompt-supplied data from its own already-generated tokens; once it has emitted a claim, later tokens condition on that claim as established fact, letting a small early error compound into a larger one ("snowballing"). A model that misreads an image early on will then confabulate further details consistent with its own wrong premise rather than reconsidering it.
  • Mismatched internal knowledge — instruction-tuning trains a model to mimic human-labeler-written responses that may draw on knowledge the labelers had but the model itself never learned, teaching it to produce confident answers even where its own knowledge is thin. Reinforcement learning from human feedback does not reliably fix this: labelers rate pairwise preference without seeing why one answer is better, giving the model no explicit signal to distinguish "I know this" from "I don't."

These accounts are complementary, not competing: self-delusion locates the cause in the autoregressive generation process itself, mismatched knowledge locates it in what training taught the model to claim.

When building factual-consistency evaluation suites, concentrate cases on queries the model is most likely to invent answers for: niche knowledge (entities and facts under-represented in training and retrieval corpora) and questions about things that do not exist (asking what someone said about a topic they never addressed). Those two patterns produce more hallucinations than well-covered, existentially valid queries, so an eval drawn only from popular facts will understate risk.

Mitigation does not come from asking the model to try harder — it comes from making claims independently checkable. Prefer completions that carry verifiable structure: an explanation of reasoning, a calculation, a source link, or specific keywords that can be searched, over a bare unqualified assertion. Contrast "there was an English king who married his cousin" (hard to check) with "...namely George IV, who married Caroline of Brunswick" (checkable). Shorter responses also fabricate less simply because there are fewer tokens in which to fabricate. The best available posture: "trust but verify, minus the trust" — treat every claim as needing independent confirmation, not as innocent until proven false. Two cheap checks help:

  • Form your own answer first. Settle on an answer before you read the model's, so its fluency doesn't anchor you (me-first collaboration).
  • Re-ask the question differently. An answer that drifts when the question is rephrased was not well grounded in the first place.

Scale the depth of verification to how opaque the system is and how much rides on the claim. Context grounding attacks the same problem from the supply side, by giving the model the facts it needs so it has less occasion to invent them, and argument hallucination is the specific case of this failure mode inside tool-call arguments.

Hallucination also compounds inconsistency: because generation is probabilistic, the same prompt can hallucinate on one call and not on another, which is part of why hallucination and inconsistency are usually discussed together as the two faces of a model's probabilistic output.

Don't assume instruction-tuning or RLHF reliably fixes this: reported findings are mixed, and at least one widely cited study found RLHF training made hallucination measurably worse relative to supervised fine-tuning alone, even though human raters still preferred the RLHF model's answers overall on other dimensions. Fixing the reward signal itself — training a reward model to explicitly penalize fabrication, not just rank pairwise preferences without explaining why one answer beats another — is a proposed but not yet standard fix. The practical stance stays the same regardless of how a given model was trained: treat every unverified claim as a candidate fabrication, not as innocent until proven false.

Sources

  • Prompt Engineering for LLMs (John Berryman), ch. 2
  • AI Engineering (Chip Huyen), ch. 2
  • AI Engineering (Chip Huyen), ch. 4
  • Taking Testing Seriously (James Bach, Michael Bolton), ch. 7