Efficient semantic uncertainty quantification in language models via diversity-steered sampling
Accurately estimating semantic aleatoric and epistemic uncertainties in large language models (LLMs) is particularly challenging in free-form question answering (QA), where obtaining stable estimates often requires many expensive generations. We introduce a diversity-steered sampler that discourages semantically redundant outputs during decoding, covers both autoregressive and masked diffusion paradigms, and yields substantial sample-efficiency gains. The key idea is to inject a continuous semantic-similarity penalty into the model's proposal distribution using a natural language inference (NLI) model lightly finetuned on partial prefixes or intermediate diffusion states. We debias downstream uncertainty estimates with importance reweighting and shrink their variance with control variates. Across four QA benchmarks, our method matches or surpasses baselines while covering more semantic clusters with the same number of samples. Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation in risk-sensitive model deployments.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceQuestion AnsweringSimilar Papers 제목 키워드 기반
Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models
Large language models (LLMs) have transformed natural language processing, but their reliable deployment requires effective uncertainty quantification (UQ). Existing UQ methods are often heuristic and lack a probabilisti…
DiversitySemantic SimilaritySemantic Textual SimilarityUncertainty QuantificationUncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume
Despite their capabilities, Multimodal Large Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries …
Audio GenerationSemantic Self-Distillation for Language Model Uncertainty
Large language models present challenges for principled uncertainty quantification, in part due to their complexity and the diversity of their outputs. Semantic dispersion, or the variance in the meaning of sampled answe…
Answer SelectionKernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities
Uncertainty quantification in Large Language Models (LLMs) is crucial for applications where safety and reliability are important. In particular, uncertainty can be used to improve the trustworthiness of LLMs by detectin…
Text GenerationUncertainty QuantificationESI: Epistemic Uncertainty Quantification via Semantic-preserving Intervention for Large Language Models
Uncertainty Quantification (UQ) is a promising approach to improve model reliability, yet quantifying the uncertainty of Large Language Models (LLMs) is non-trivial. In this work, we establish a connection between the un…
Computational Efficiency