Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning
Reward baseline is important for Reinforcement Learning (RL) algorithms to reduce variance in policy gradient estimates. Recently, for language modeling, Group Relative Policy Optimization (GRPO) is proposed to compute the advantage for each output by subtracting the mean reward, as the baseline, for all outputs in the group. However, it can lead to inaccurate advantage estimates in environments with highly noisy rewards, potentially introducing bias. In this work, we propose a model, called Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), by using lightweight Kalman filtering to dynamically estimate the latent reward mean and variance. This filtering technique replaces the naive batch mean baseline, enabling more adaptive advantage normalization. Our method does not require additional learned parameters over GRPO. This approach offers a simple yet effective way to incorporate multiple outputs of GRPO into advantage estimation, improving policy optimization in settings where highly dynamic reward signals are difficult to model for language models. Through accuracy and rewards obtained from math question answering and reasoning, we show that using a more adaptive advantage estimation model, KRPO can improve the stability and performance of GRPO. The code is available at https://github.com/billhhh/KRPO_LLMs_RL.
Code (1)
Tasks
Language ModelingLanguage ModellingMathQuestion AnsweringReinforcement Learning (RL)Similar Papers 제목 키워드 기반
EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
Reinforcement learning (RL) for LLM post-training faces a fundamental design choice: whether to use a learned critic as a baseline for policy optimization. Classical theory favors critic-based methods such as PPO for var…
Reinforcement LearningMathematical ReasoningDeep Reinforcement Learning-Based Beam Tracking for Low-Latency Services in Vehicular Networks
Ultra-Reliable and Low-Latency Communications (URLLC) services in vehicular networks on millimeter-wave bands present a significant challenge, considering the necessity of constantly adjusting the beam directions. Conven…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)NDKF: A Neural-Enhanced Distributed Kalman Filter for Nonlinear Multi-Sensor Estimation
We propose a Neural-Enhanced Distributed Kalman Filter (NDKF) for multi-sensor state estimation in nonlinear systems. Unlike traditional Kalman filters that rely on explicit, linear models and centralized data fusion, th…
State EstimationK-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning
We propose a simple yet effective alternative to reward normalization in policy gradient reinforcement learning by integrating a 1D Kalman filter for online reward estimation. Instead of relying on fixed heuristics, our …
Reinforcement LearningShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collect…
Reinforcement Learning