Hallucinations persist because benchmarks reward confident guessing

Why do LLMs keep getting things confidently wrong when saying "I don't know" would serve everyone better? Researchers at OpenAI offer a clear answer: the root of the problem is statistical. It appears during pretraining and is then locked in by the way we evaluate models after fine-tuning. In short: the data can be free of errors and the training objective will still push the model toward guessing — which is exactly what optimizing cross-entropy does. Then the benchmarks take over, and since none of them award points for "I don't know," guessing becomes the winning strategy.

Start with the basics

Imagine reducing text generation to a simple question: valid or invalid? If a model cannot reliably tell valid answers from false ones, it will inevitably produce false ones. The authors prove the link rigorously: the generative error rate is at least about twice the error rate of that binary classifier. Producing text is harder than answering yes or no — but if the model already stumbles at the yes/no level, generation will go wrong more often still.

What matters most: even if the corpus contains no explicit errors whatsoever, many facts are arbitrary — there is no pattern in them to infer from context. Birthdays, for example. If some share of such facts appears exactly once in training, the base estimate says the hallucination rate on those questions will be no lower than the share of singletons. The result rests on the missing-mass idea from classical statistics: what you have never seen, or seen only once, is what you will get wrong most often. The authors adapt the missing-mass estimate to this setting and derive a workable lower bound on errors for arbitrary facts. The advice that follows is straightforward: don't confuse memory with understanding, and give evaluations a legitimate path to "I don't know."

On prompts

In practice a model answers contextual questions, where several answers can be valid and the wrong ones are many. The formal analysis holds: under reasonable calibration, the probability of generative error is bounded from below by the quality of the classifier and by the ratio between the number of valid and invalid answers for each prompt. A poor model — limited representational capacity, thin context, tokenization, constrained resources — raises that floor. Letter counting is a telling example: some LLMs miscount the letters in a word because of tokenization, while others work through the problem step by step and cut the error rate sharply.

Post-training

Why don't RLHF and related techniques remove hallucinations? Because most mainstream benchmarks use binary metrics with no credit for an appropriate refusal: a correct answer scores 1, a wrong one 0, and "I don't know" also 0. Under that scoring the optimal strategy is to guess whenever there is any chance at all of landing on the right option. What wins is not the most reliable model but the most aggressive guesser. As long as the leaderboards run on metrics like these, the incentive to behave safely stays weak.

The authors propose a simple change to the rules: write an explicit confidence threshold t into the instructions. Confidence above t — answer; below it — say "I don't know." And penalize a wrong answer more heavily than a refusal: the penalty is t/(1−t). The optimal strategy then becomes the right one — answer only where the probability of being correct is genuinely high. This can be dropped into existing benchmarks without building new special-purpose tests, with models compared across several values of t. It also opens a way to audit behavioral calibration: watch how the answer rate and the accuracy shift as the threshold rises.

The paper's practical observations fit this picture well. Different LLMs confidently produce different wrong facts about dissertations and birth dates. On simple counting questions they disagree by whole integers. None of these errors looks like an anomaly — by this analysis, they are what current training and evaluation objectives predict.

What won't fix itself

Search and retrieval-augmented generation do help, but not always: when the sources don't settle the question, or the task is hard enough — ciphers, fine-grained counting, ambiguous terms — the incentive to guess survives any binary metric. Open-ended tasks such as biographies only raise the risk: a single noticeable inaccuracy is already an error. Here it makes sense to measure the degree of hallucination by the number of factual misses and to reward a correct refusal to supply details.

The conclusion is simple: hallucinations are a natural product of modeling text, and they are reinforced by how we count points in benchmarks. To see real progress, the metrics need one small turn of the dial — reward well-placed uncertainty. Then LLMs will be not only strong across tests but dependable assistants that honestly choose to stay quiet on a question when confidence is low.