ML//hallucination

A hallucination is content a model generates with confidence that is plausible but factually wrong. It is intrinsic when it contradicts the source or the context, and extrinsic when it makes up facts that are in no source.

A hallucination is content a model generates with confidence that is plausible but factually wrong. It is intrinsic when it contradicts the source or the context, and extrinsic when it makes up facts that are in no source.

Its root causes are noise in the training data, pattern matching taking the place of factual recall, and a language modeling objective that optimizes plausibility rather than truth. The problem is unsolved: RAG (built mainly for this, grounding generation in retrieved documents so the model has facts to cite instead of inventing them), citations and confidence calibration mitigate it, and none of them eliminates it.

Where an answer can be checked, the system can catch what the model cannot. Code can be run against tests, a calculation recomputed by a tool, a claimed relation looked up in a knowledge graph; a verifier turns a confident error into a rejected candidate. The confidence a model states is a poor guide here, because it is often poorly calibrated: a model saying it is sure is not evidence that it is right.