Sample Efficient Reinforcement Learning with REINFORCE

2020-10-22 · Junzi Zhang, Jongho Kim, Brendan O'Donoghue, Stephen Boyd

Policy gradient methods are among the most effective methods for large-scale reinforcement learning, and their empirical success has prompted several works that develop the foundation of their global convergence theory. However, prior works have either required exact gradients or state-action visitation measure based mini-batch stochastic gradients with a diverging batch size, which limit their applicability in practical scenarios. In this paper, we consider classical policy gradient methods that compute an approximate gradient with a single trajectory or a fixed size mini-batch of trajectories under soft-max parametrization and log-barrier regularization, along with the widely-used REINFORCE gradient estimation procedure. By controlling the number of "bad" episodes and resorting to the classical doubling trick, we establish an anytime sub-linear high probability regret bound as well as almost sure global convergence of the average regret with an asymptotically sub-linear rate. These provide the first set of global convergence and sample efficiency results for the well-known REINFORCE algorithm and contribute to a better understanding of its performance in practice.

📄 PDF Abstract BibTeX arXiv:2010.11364

Code (0)

등록된 구현이 없습니다.

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

REINFORCE REINFORCE is a Monte Carlo variant of a policy gradient algorithm in reinforcement learning. The agent collects samples of an episode using its current policy, and uses it to…

Episodic Reinforcement Learning with Associative Memory

2020-05-01 · ICLR 2020 1 · Guangxiang Zhu*, Zichuan Lin*, Guangwen Yang, Chongjie Zhang

Sample efficiency has been one of the major challenges for deep reinforcement learning. Non-parametric episodic control has been proposed to speed up parametric reinforcement learning by rapidly latching on previously su…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Rating-based Reinforcement Learning

2023-07-30 · Devin White, Mingkang Wu, Ellen Novoseller, Vernon J. Lawhern 외

This paper develops a novel rating-based reinforcement learning approach that uses human ratings to obtain human guidance in reinforcement learning. Different from the existing preference-based and ranking-based reinforc…

reinforcement-learning Reinforcement Learning

Mask-based Predictive Representations for Reinforcement Learning

2026-07-05 · Kai Zhao arxiv

Vision-based deep reinforcement learning involves dealing with high-dimensional inputs of image information. It is crucial to abstract effective states from high-dimensional image inputs and limited samples for sample-ef…

Reinforcement Learning

Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning

2020-03-10 · Stephanie Milani, Nicholay Topin, Brandon Houghton, William H. Guss 외

To facilitate research in the direction of sample efficient reinforcement learning, we held the MineRL Competition on Sample Efficient Reinforcement Learning Using Human Priors at the Thirty-third Conference on Neural In…

Deep Reinforcement Learning Imitation Learning reinforcement-learning Reinforcement Learning+1

Recruitment-imitation Mechanism for Evolutionary Reinforcement Learning

2019-12-13 · Shuai Lü, Shuai Han, Wenbo Zhou, Junwei Zhang

Reinforcement learning, evolutionary algorithms and imitation learning are three principal methods to deal with continuous control tasks. Reinforcement learning is sample efficient, yet sensitive to hyper-parameters sett…

continuous-control Continuous Control Efficient Exploration Evolutionary Algorithms+5