홈 › Papers

Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments

2025-11-05 · Bryan L. M. de Oliveira, Felipe V. Frujeri, Marcos P. C. M. Queiroz, Luana G. B. Martins, Telma W. de L. Soares, Luckeciano C. Melo arxiv

Group Relative Policy Optimization (GRPO) has emerged as a scalable alternative to Proximal Policy Optimization (PPO) by eliminating the learned critic and instead estimating advantages through group-relative comparisons of trajectories. This simplification raises fundamental questions about the necessity of learned baselines in policy-gradient methods. We present the first systematic study of GRPO in classical single-task reinforcement learning environments, spanning discrete and continuous control tasks. Through controlled ablations isolating baselines, discounting, and group sampling, we reveal three key findings: (1) learned critics remain essential for long-horizon tasks: all critic-free baselines underperform PPO except in short-horizon environments like CartPole where episodic returns can be effective; (2) GRPO benefits from high discount factors (gamma = 0.99) except in HalfCheetah, where lack of early termination favors moderate discounting (gamma = 0.9); (3) smaller group sizes outperform larger ones, suggesting limitations in batch-based grouping strategies that mix unrelated episodes. These results reveal both the limitations of critic-free methods in classical control and the specific conditions where they remain viable alternatives to learned value functions.

📄 PDF Abstract BibTeX arXiv:2511.03527

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Continuous Control

Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training

2025-05-28 · Youssef Mroueh, Nicolas Dupuis, Brian Belgodere, Apoorva Nitsure 외

We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training …

Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning

2025-05-21 · Yurun Yuan, Fan Chen, Zeyu Jia, Alexander Rakhlin 외

Policy-based methods currently dominate reinforcement learning (RL) pipelines for large language model (LLM) reasoning, leaving value-based approaches largely unexplored. We revisit the classical paradigm of Bellman Resi…

Language Modeling Language Modelling Large Language Model Mathematical Reasoning+1

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

2019-03-11 · Denis Steckelmacher, Hélène Plisnier, Diederik M. Roijers, Ann Nowé

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Thompson Sampling

Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoning

2025-10-02 · Jiashun Liu, Johan Obando-Ceron, Han Lu, Yancheng He 외 arxiv

Most recent RL for LLMs (RL4LLM) methods avoid explicit critics, replacing them with average advantage baselines. This shift is largely pragmatic: conventional value functions are computationally expensive to train at LL…

Deep Reinforcement Learning With Adaptive Combined Critics

2021-01-01 · Huihui Zhang, Wu Huang

The overestimation problem has long been popular in deep value learning, because function approximation errors may lead to amplified value estimates and suboptimal policies. There have been several methods to deal with t…

continuous-control Continuous Control Deep Reinforcement Learning reinforcement-learning+2