AI still confidently invents facts — and we can't fully make it stop

The problem

Language models sometimes generate fluent, confident statements that are simply false — a failure researchers began calling 'hallucination' in the late 2010s, and which every frontier model still exhibits. In 2024, Oxford researchers published a Nature method using 'semantic entropy' to detect a major class of hallucinations by measuring a model's uncertainty over meanings rather than words. In 2025, an OpenAI and Georgia Tech analysis argued that some hallucination is a statistical inevitability of how models are trained — and that it persists partly because benchmarks reward confident guessing over honest abstention. Retrieval grounding, uncertainty estimation, and abstention training all reduce error rates, but no known technique makes a general-purpose model reliably truthful, especially across the multi-step reasoning chains agents now execute.

Why it matters

These systems now draft legal and medical text, answer search queries, and take autonomous actions where a single invented fact can compound into real-world harm. Reliable self-knowledge — an AI that can be trusted to say 'I don't know' — is arguably the gating requirement for delegating consequential work to machines.

Progress so far

• 2018hallucination is identified and named as a failure mode in neural text generation
• 2022systematic surveys establish hallucination as a central open problem of natural language generation
• 2024Farquhar et al. publish semantic-entropy detection of confabulations in Nature
• 2025Kalai, Nachum, Vempala, and Zhang show statistical pressures make some errors unavoidable and that guess-rewarding benchmarks entrench them

References

1. arXiv (Kalai et al., 2025) — 'Why Language Models Hallucinate' — the statistical account of why hallucination arises and persists
2. Nature (Farquhar et al., 2024) — semantic-entropy method for detecting hallucinated 'confabulations'