OpenAI 2024. "Why Language Models Hallucinate"

Research
, , , , , , ,
1

https://openai.com/index/why-language-models-hallucinate/

Overview

[AI Summary]: This research paper from OpenAI investigates the fundamental causes of hallucinations in language models, arguing that standard training and evaluation procedures inadvertently reward models for guessing rather than acknowledging uncertainty. The authors demonstrate that current accuracy-based evaluations encourage models to provide plausible but incorrect answers instead of abstaining when uncertain, and propose that penalizing confident errors more than uncertainty expressions would reduce hallucinations. The paper provides statistical analysis showing how hallucinations arise from next-word prediction during pretraining, particularly for low-frequency facts that cannot be reliably predicted from patterns alone.