Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units

2021-10-14 · Kenny Young

Training neural networks with discrete stochastic variables presents a unique challenge. Backpropagation is not directly applicable, nor are the reparameterization tricks used in networks with continuous stochastic variables. To address this challenge, we present Hindsight Network Credit Assignment (HNCA), a novel gradient estimation algorithm for networks of discrete stochastic units. HNCA works by assigning credit to each unit based on the degree to which its output influences its immediate children in the network. We prove that HNCA produces unbiased gradient estimates with reduced variance compared to the REINFORCE estimator, while the computational cost is similar to that of backpropagation. We first apply HNCA in a contextual bandit setting to optimize a reward function that is unknown to the agent. In this setting, we empirically demonstrate that HNCA significantly outperforms REINFORCE, indicating that the variance reduction implied by our theoretical analysis is significant and impactful. We then show how HNCA can be extended to optimize a more general function of the outputs of a network of stochastic units, where the function is known to the agent. We apply this extended version of HNCA to train a discrete variational auto-encoder and empirically show it compares favourably to other strong methods. We believe that the ideas underlying HNCA can help stimulate new ways of thinking about efficient credit assignment in stochastic compute graphs.

📄 PDF Abstract BibTeX arXiv:2110.07700

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

REINFORCE REINFORCE is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to…

Towards Causal Credit Assignment

2022-12-22 · Mátyás Schubert

Adequately assigning credit to actions for future outcomes based on their contributions is a long-standing open challenge in Reinforcement Learning. The assumptions of the most commonly used credit assignment method are …

counterfactual

Hindsight Credit Assignment

2019-12-05 · NeurIPS 2019 12 · Anna Harutyunyan, Will Dabney, Thomas Mesnard, Mohammad Azar 외

We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihoo…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Towards Practical Credit Assignment for Deep Reinforcement Learning

2021-06-08 · Vyacheslav Alipov, Riley Simmons-Edler, Nikita Putintsev, Pavel Kalinin 외

Credit assignment is a fundamental problem in reinforcement learning, the problem of measuring an action's influence on future rewards. Explicit credit assignment methods have the potential to boost the performance of RL…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1