Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning

2025-10-09 · Aman Sharma, Paras Chopra arxiv

We introduce a simple, yet novel entropy-based framework to drive token efficiency in large language models during reasoning tasks. Our approach uses Shannon entropy from token-level logprobs as a confidence signal to enable early stopping, achieving 25-50% computational savings while maintaining task accuracy. Crucially, we demonstrate that entropy-based confidence calibration represents an emergent property of advanced post-training optimization present in modern reasoning models but notably absent in standard instruction-tuned and pre-trained models (Llama 3.3 70B). We show that the entropy threshold to stop reasoning varies from model to model but can be calculated easily in one shot using only a few examples from existing reasoning datasets. Our results indicate that advanced reasoning models often know that they've gotten a correct answer early on, and that this emergent confidence awareness can be exploited to save tokens and reduce latency. The framework demonstrates consistent performance across reasoning-optimized model families with 25-50% computational cost reduction while preserving accuracy, revealing that confidence mechanisms represent a distinguishing characteristic of modern post-trained reasoning systems versus their predecessors.

📄 PDF Abstract BibTeX arXiv:2510.08146

Code (0)

등록된 구현이 없습니다.

DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Models

2026-06-22 · Jungseob Lee, Seongtae Hong, Seungjun Lee, Jaehyung Seo 외 arxiv

Hybrid reasoning models can answer directly or spend extra tokens on extended thinking. A practical router should choose between these modes for each query, so easy problems avoid unnecessary reasoning and hard problems …

Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs

2026-01-29 · Lei Yang, Wei Bi, Chenxi Sun, Renren Jin 외 arxiv

On-policy reinforcement learning (RL) for language model post-training suffers from a fundamental tension: as training progresses, policy entropy collapses and sampling diversity diminishes, causing the model to ``forget…

Mathematical Reasoning Reinforcement Learning

Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning

2026-05-13 · Junlong Ke, Zichen Wen, Weijia Li, Conghui He 외 arxiv

On-policy self-distillation trains a reasoning model on its own rollouts while a teacher, often the same model conditioned on privileged context, provides dense token-level supervision. Existing objectives typically weig…

Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Nature

2025-09-20 · Zheng Liu, Mengjie Liu, Siwei Wen, Mengzhang Cai 외 arxiv

Using entropy as a measure of heterogeneity to guide optimization has emerged as a crucial research direction in Reinforcement Learning for LLMs. However, existing methods typically treat it as a discrete filter or post-…

Reinforcement Learning Mathematical Reasoning

Rethinking Reverse KL as Adaptive Entropy Distillation

2026-08-05 · Shizhen Li, Zhiyu Shen, Yuyin Lu, Yunhe Pang 외 arxiv

Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In …

Mathematical Reasoning Knowledge Distillation