EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity

2025-07-29 · Xingjian Zhang, Siwei Wen, Wenjun Wu, Lei Huang arxiv

Large Language Models (LLMs) have made remarkable progress in enhancing step-by-step reasoning through reinforcement learning. However, the Group Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward rules, often encounters the issue of identical rewards within groups, leading to the advantage collapse problem. Existing works typically address this challenge from two perspectives: enforcing model reflection to enhance response diversity, and introducing internal feedback to augment the training signal (advantage). In this work, we begin by analyzing the limitations of model reflection and investigating the policy entropy of responses at the fine-grained sample level. Based on our experimental findings, we propose the EDGE-GRPO algorithm, which adopts \textbf{E}ntropy-\textbf{D}riven Advantage and \textbf{G}uided \textbf{E}rror Correction to effectively mitigate the problem of advantage collapse. Extensive experiments on several main reasoning benchmarks demonstrate the effectiveness and superiority of our approach. It is available at https://github.com/ZhangXJ199/EDGE-GRPO.

📄 PDF Abstract BibTeX arXiv:2507.21848

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

EP-GRPO: Entropy-Progress Aligned Group Relative Policy Optimization with Implicit Process Guidance

2026-05-06 · Song Yu, Li Li, Wenwen Zhao, Zhisheng Yang arxiv

Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has advanced LLM reasoning. However, GRPO suffers from three credit assignment failures: uniform token-level …

Reinforcement Learning Mathematical Reasoning

From Broad Exploration to Stable Synthesis: Entropy-Guided Optimization for Autoregressive Image Generation

2026-03-12 · Han Song, Yucheng Zhou, Jianbing Shen, Yu Cheng arxiv

Combining Chain-of-Thought (CoT) with Reinforcement Learning (RL) improves text-to-image (T2I) generation, yet the underlying interaction between CoT's exploration and RL's optimization remains unclear. We present a syst…

Reinforcement Learning Image Generation

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

2025-05-18 · Minghan Chen, Guikun Chen, Wenguan Wang, Yi Yang

Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variati…

Math Mathematical Reasoning

Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models

2026-06-03 · Chirag Chawla, Rohan Charudatt Salvi, Madhav S. Baidya arxiv

Group Relative Policy Optimisation (GRPO) has emerged as an effective reinforcement-learning algorithm for aligning language models on reasoning tasks, but it treats every token position and every sampled rollout symmetr…

Reinforcement Learning Mathematical Reasoning

SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR

2026-06-16 · Siddharth Aphale, Kelly Liu arxiv

The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)…