Efficient Process Reward Model Training via Active Learning

2025-04-14 · Keyu Duan, Zichen Liu, Xin Mao, Tianyu Pang, Changyu Chen, Qiguang Chen, Michael Qizhe Shieh, Longxu Dou

Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss with respect to the labels and update the PRM's weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduces 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60% of the data. A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same sized models.

📄 PDF Abstract BibTeX arXiv:2504.10559

Code (1)

sail-sg/activeprm 공식 구현 pytorch

Tasks

Active LearningMath

Active Inverse Reward Design

2018-09-09 · Sören Mindermann, Rohin Shah, Adam Gleave, Dylan Hadfield-Menell

Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring thi…

Informativeness

Risk-averse Batch Active Inverse Reward Design

2023-11-20 · Panagiotis Liampas

Designing a perfect reward function that depicts all the aspects of the intended behavior is almost impossible, especially generalizing it outside of the training environments. Active Inverse Reward Design (AIRD) propose…

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

2021-06-09 · Kimin Lee, Laura Smith, Pieter Abbeel

Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop R…

reinforcement-learning Reinforcement Learning (RL)Unsupervised Pre-training

Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

2026-07-28 · Chenrui Shi, Yuwei Wu, Yang Liu, Ruining Feng 외 arxiv

Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction. Automated GUI task evaluation has received increasing attention because the evaluation results…

Reinforcement Learning

RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

2026-07-06 · Qiang Liu, Taian Guo, Ruizhi Qiao, Xing Sun arxiv

Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, dir…