홈 › Papers

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

2020-11-18 · Thomas Mesnard, Théophane Weber, Fabio Viola, Shantanu Thakoor, Alaa Saade, Anna Harutyunyan, Will Dabney, Tom Stepleton, Nicolas Heess, Arthur Guez, Éric Moulines, Marcus Hutter, Lars Buesing, Rémi Munos

Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. In particular, this requires separating skill from luck, i.e. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We formulate a family of policy gradient algorithms that use these future-conditional value functions as baselines or critics, and show that they are provably low variance. To avoid the potential bias from conditioning on future information, we constrain the hindsight information to not contain information about the agent's actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative and challenging problems.

📄 PDF Abstract BibTeX arXiv:2011.09464

Code (0)

등록된 구현이 없습니다.

Tasks

counterfactualmodelreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Counterfactuals 설명 없음

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

2026-03-23 · Zhongyi Li, Wan Tian, Jinju Chen, Huiming Zhang 외 arxiv

Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure …

Reinforcement Learning Mathematical Reasoning

Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

2023-09-21 · NeurIPS 2023 11

To make reinforcement learning more sample efficient, we need better credit assignment methods that measure an action’s influence on future rewards. Building upon Hindsight Credit Assignment (HCA), we introduce Counterfa…

Model-Free Counterfactual Credit Assignment

2021-01-01 · Thomas Mesnard, Theophane Weber, Fabio Viola, Shantanu Thakoor 외

Credit assignment in reinforcement learning is the problem of measuring an action’s influence on future rewards. In particular, this requires separating \emph{skill} from \emph{luck}, ie.\ disentangling the effect of an…

counterfactual model valid

Selective Credit Assignment

2022-02-20 · Veronica Chelu, Diana Borsa, Doina Precup, Hado van Hasselt

Efficient credit assignment is essential for reinforcement learning algorithms in both prediction and control settings. We describe a unified view on temporal-difference algorithms for selective credit assignment. These …

Prediction reinforcement-learning Reinforcement Learning (RL)

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

2021-06-01 · Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu 외

Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challe…

counterfactual Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3