Selective Credit Assignment
Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings. We describe a unified view on temporal-difference algorithms for selective credit assignment. These selective algorithms apply weightings to quantify the contribution of learning updates. We present insights into applying weightings to value-based learning and planning algorithms, and describe their role in mediating the backward credit distribution in prediction and control. Within this space, we identify some existing online learning algorithms that can assign credit selectively as special cases, as well as add new algorithms that assign credit backward in time counterfactually, allowing credit to be assigned off-trajectory and off-policy.
Code (0)
등록된 구현이 없습니다.
Tasks
Predictionreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR
Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Op…
Reinforcement LearningAn Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics wo…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Towards Causal Credit Assignment
Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are …
counterfactualLearning Explicit Credit Assignment for Multi-agent Joint Q-learning
Multi-agent joint Q-learning based on Centralized Training with Decentralized Execution (CTDE) has become an effective technique for multi-agent cooperation. During centralized training, these methods are essentially add…
Q-LearningRevisiting QMIX: Discriminative Credit Assignment by Gradient Entropy Regularization
In cooperative multi-agent systems, agents jointly take actions and receive a team reward instead of individual rewards. In the absence of individual reward signals, credit assignment mechanisms are usually introduced to…