Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration

2025-11-02 · Yan Sun, Jia Guo, Stanley Kok, Zihao Wang, Zujie Wen, Zhiqiang Zhang arxiv

Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning ability of large language models, yet training remains costly because many rollouts contribute little to optimization, considering the amount of computation required. This study investigates how simply leveraging intrinsic data properties, almost free benefit during training, can improve data efficiency for RLVR. We propose PREPO with two complementary components. First, we adopt prompt perplexity as an indicator of model adaptability in learning, enabling the model to progress from well-understood contexts to more challenging ones. Second, we amplify the discrepancy among the rollouts by differentiating their relative entropy, and prioritize sequences that exhibit a higher degree of exploration. Together, these mechanisms reduce rollout demand while preserving competitive performance. On the Qwen and Llama models, PREPO achieves effective results on mathematical reasoning benchmarks with up to 3 times fewer rollouts than the baselines. Beyond empirical gains, we provide theoretical and in-depth analyses explaining the underlying rationale of our method to improve the data efficiency of RLVR.

📄 PDF Abstract BibTeX arXiv:2511.00794

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Mathematical Reasoning

Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Tasks

2023-09-28 · Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li

Current reinforcement learning algorithms struggle in sparse and complex environments, most notably in long-horizon manipulation tasks entailing a plethora of different sequences. In this work, we propose the Intrinsical…

Decision Making reinforcement-learning Reinforcement Learning

Improving Intrinsic Exploration with Language Abstractions

2022-02-17 · Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang 외

Reinforcement learning (RL) agents are particularly hard to train when rewards are sparse. One common solution is to use intrinsic rewards to encourage agents to explore their environment. However, recent intrinsic explo…

reinforcement-learning Reinforcement Learning (RL)

Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards

2025-10-18 · Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li 외 arxiv

Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limit…

Reinforcement Learning

Application of LLMs to Multi-Robot Path Planning and Task Allocation

2025-07-09 · Ashish Kumar arxiv

Efficient exploration is a well known problem in deep reinforcement learning and this problem is exacerbated in multi-agent reinforcement learning due the intrinsic complexities of such algorithms. There are several appr…

Multi-agent Reinforcement Learning

Intrinsically Guided Exploration in Meta Reinforcement Learning

2021-01-01 · Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen 외

Deep reinforcement learning algorithms generally require large amounts of data to solve a single task. Meta reinforcement learning (meta-RL) agents learn to adapt to novel unseen tasks with high sample efficiency by extr…

Deep Reinforcement Learning Efficient Exploration Meta Reinforcement Learning MuJoCo+3