Hindsight Network Credit Assignment

2020-11-24 · Kenny Young

We present Hindsight Network Credit Assignment (HNCA), a novel learning method for stochastic neural networks, which works by assigning credit to each neuron's stochastic output based on how it influences the output of its immediate children in the network. We prove that HNCA provides unbiased gradient estimates while reducing variance compared to the REINFORCE estimator. We also experimentally demonstrate the advantage of HNCA over REINFORCE in a contextual bandit version of MNIST. The computational complexity of HNCA is similar to that of backpropagation. We believe that HNCA can help stimulate new ways of thinking about credit assignment in stochastic compute graphs.

📄 PDF Abstract BibTeX arXiv:2011.12351

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

REINFORCE REINFORCE is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to…

Towards Causal Credit Assignment

2022-12-22 · Mátyás Schubert

Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are …

counterfactual

Hindsight Credit Assignment

2019-12-05 · NeurIPS 2019 12 · Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Azar 외

We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihoo…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Towards Practical Credit Assignment for Deep Reinforcement Learning

2021-06-08 · Vyacheslav Alipov, Riley Simmons-Edler, Nikita Putintsev, Pavel Kalinin 외

Credit assignment is a fundamental problem in reinforcement learning, the problem of measuring an action's influence on future rewards. Explicit credit assignment methods have the potential to boost the performance of RL…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Variance Reduced Advantage Estimation with $δ$ Hindsight Credit Assignment

2019-11-19 · Kenny Young

Hindsight Credit Assignment (HCA) refers to a recently proposed family of methods for producing more efficient credit assignment in reinforcement learning. These methods work by explicitly estimating the probability that…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning

2023-07-21 · Akash Velu, Skanda Vaidyanath, Dilip Arumugam

Oftentimes, environments for sequential decision-making problems can be quite sparse in the provision of evaluative feedback to guide reinforcement-learning agents. In the extreme case, long trajectories of behavior are …

Decision Making Deep Reinforcement Learning Off-policy evaluation Policy Gradient Methods+3