From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction
In this work, we study the credit assignment problem in reward augmented maximum likelihood (RAML) learning, and establish a theoretical equivalence between the token-level counterpart of RAML and the entropy regularized reinforcement learning. Inspired by the connection, we propose two sequence prediction algorithms, one extending RAML with fine-grained credit assignment and the other improving Actor-Critic with a systematic entropy regularization. On two benchmark datasets, we show the proposed algorithms outperform RAML and Actor-Critic respectively, providing new alternatives to sequence prediction.
Code (1)
Tasks
Predictionreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Revisiting QMIX: Discriminative Credit Assignment by Gradient Entropy Regularization
In cooperative multi-agent systems, agents jointly take actions and receive a team reward instead of individual rewards. In the absence of individual reward signals, credit assignment mechanisms are usually introduced to…
Guided Adaptive Credit Assignment for Sample Efficient Policy Optimization
Policy gradient methods have achieved remarkable successes in solving challenging reinforcement learning problems. However, it still often suffers from sparse reward tasks, which leads to poor sample efficiency during tr…
Instruction FollowingPolicy Gradient MethodsGTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all toke…
Reinforcement LearningBeyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Op…
Reinforcement LearningLearning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
We present a multi-agent actor-critic method that aims to implicitly address the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment among agents may not require an ex…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1