Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding
In open-ended generation, LLMs frequently fall into the "likelihood trap", marked by repetitive degeneration and vocabulary dullness, creating a discrepancy between machine-generated and human-written text. While post-hoc tail truncation (e.g., Top-$p$, Min-$p$) avoids sampling from the unreliable tail, it can over-sample from the uncalibrated head and misalign generation with human lexical preferences; fixed scalar repetition penalties likewise ignore variation in logit scale across inference steps, potentially disrupting semantic coherence. To address both limitations, we propose Variance-Calibrated Modulation (VCM), a training-free pre-decoding intervention that reshapes the probability distribution before truncation through two dynamic mechanisms: (1) Contextual Searchlight via PMI, which suppresses global stopwords while elevating context-evoked tokens, and (2) Adaptive Self-Debiasing, which uses real-time logit standard deviation for scale-invariant penalization. Across open-ended generation, factual QA, and mathematical reasoning, VCM consistently mitigates the likelihood trap. With negligible computational overhead, VCM integrates with existing decoding strategies, improving diversity, coherence, and, particularly at higher decoding temperatures, reasoning accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Mathematical ReasoningSimilar Papers 제목 키워드 기반
Ribbon: Scalable Approximation and Robust Uncertainty Quantification
Reliably quantifying predictive uncertainty is difficult for complex, high-dimensional, or misspecified models. Both fully Bayesian and bootstrap resampling methods provide principled uncertainty estimates but are often …
Sampling-Free Learning of Bayesian Quantized Neural Networks
Bayesian learning of model parameters in neural networks is important in scenarios where estimates with well-calibrated uncertainty are important. In this paper, we propose Bayesian quantized networks (BQNs), quantized n…
image-classificationImage ClassificationNovel LoS $β-γ$ THz Channel Unifying Molecular Re-radiation Manifestations
This paper introduces a novel line-of-sight (LoS) $\beta-\gamma$ terahertz (THz) channel model that closely mirrors physical reality by considering radiation trapping. Our channel model provides an exhaustive modeling of…
Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Families
Many differentially private (DP) data release systems either output DP synthetic data and leave analysts to perform inference as usual, which can lead to severe miscalibration, or output a DP point estimate without a pri…
Synthetic Data GenerationClassification of Radio Signals Using Truncated Gaussian Discriminant Analysis of Convolutional Neural Network-Derived Features
To improve the utility and scalability of distributed radio frequency (RF) sensor and communication networks, reduce the need for convolutional neural network (CNN) retraining, and efficiently share learned information a…
ClassificationGeneral Classification