GPG: Generalized Policy Gradient Theorem for Transformer-based Policies
We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies. Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework. Furthermore, we explore its practical applications in training Large Language Models (LLMs), offering new insights into efficient policy optimization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Expected Policy Gradients for Reinforcement Learning
We propose expected policy gradients (EPG), which unify stochastic policy gradients (SPG) and deterministic policy gradients (DPG) for reinforcement learning. Inspired by expected sarsa, EPG integrates (or sums) across a…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Generalized Off-Policy Actor-Critic
We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excurs…
counterfactualMuJoCoreinforcement-learningReinforcement Learning+1Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates
The policy gradient theorem states that the policy should only be updated in states that are visited by the current policy, which leads to insufficient planning in the off-policy states, and thus to convergence to subopt…
Dr Jekyll & Mr Hyde: the strange case of off-policy policy updates
The policy gradient theorem states that the policy should only be updated in states that are visited by the current policy, which leads to insufficient planning in the off-policy states, and thus to convergence to subopt…
Policy Gradient for Reinforcement Learning with General Utilities
In Reinforcement Learning (RL), the goal of agents is to discover an optimal policy that maximizes the expected cumulative rewards. This objective may also be viewed as finding a policy that optimizes a linear function o…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)