Improving Uncertainty Estimation through Semantically Diverse Language Generation

Large language models (LLMs) can suffer from hallucinations when generating
text. These hallucinations impede various applications in society and industry
by making LLMs untrustworthy. Current LLMs generate text in an autoregres-
sive fashion by predicting and appending text tokens. When an LLM is uncertain
about the semantic meaning of the next tokens to generate, it is likely to start
hallucinating. Thus, it has been suggested that predictive uncertainty is one of
the main causes of hallucinations. We introduce Semantically Diverse Language
Generation (SDLG) to quantify predictive uncertainty in LLMs. SDLG steers
the LLM to generate semantically diverse yet likely alternatives for an initially
generated text. This approach provides a precise measure of aleatoric semantic
uncertainty, detecting whether the initial text is likely to be hallucinated. Exper-
iments on question-answering tasks demonstrate that SDLG consistently outper-
forms existing methods while being the most computationally efficient, setting a
new standard for uncertainty estimation in LLMs.