ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models

2025-05-22 · Razvan-Gabriel Dumitru, Darius Peteleaza, Vikas Yadav, Liangming Pan

Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency-accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.

📄 PDF Abstract BibTeX arXiv:2505.17250

Code (1)

razvandu/conciserl 공식 구현 pytorch

Tasks

Large Language ModelMathreinforcement-learning Reinforcement Learning

Efficient Reasoning via Reward Model

2025-11-12 · Yuhao Wang, Xiaopeng Li, Cheng Gong, Ziru Liu 외 arxiv

Reinforcement learning with verifiable rewards (RLVR) has been shown to enhance the reasoning capabilities of large language models (LLMs), enabling the development of large reasoning models (LRMs). However, LRMs such as…

Reinforcement Learning

Training Large Language Models to Reason via EM Policy Gradient

2025-04-24 · Tianbing Xu

Recently, foundation models such as OpenAI's O1 and O3, along with DeepSeek's R1, have demonstrated strong reasoning capacities and problem-solving skills acquired through large-scale reinforcement learning (RL), with wi…

GSM8K Math Problem Decomposition reinforcement-learning+2

Think When You Need: Self-Adaptive Chain-of-Thought Learning

2025-04-04 · Junjie Yang, Ke Lin, Xing Yu

Chain of Thought (CoT) reasoning enhances language models' performance but often leads to inefficient "overthinking" on simple problems. We identify that existing approaches directly penalizing reasoning length fail to a…

Concise Reasoning via Reinforcement Learning

2025-04-07 · Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, Kartik Talamadupula

Despite significant advancements in large language models (LLMs), a major drawback of reasoning models is their enormous token usage, which increases computational cost, resource requirements, and response time. In this …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models

2025-10-04 · Canhui Wu, Qiong Cao, Chang Li, Zhenfang Wang 외 arxiv

Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize gene…