Variance Reduction for Reinforcement Learning in Input-Driven Environments

2018-07-06 · ICLR 2019 5 · Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, Mohammad Alizadeh

We consider reinforcement learning in input-driven environments, where an exogenous, stochastic input process affects the dynamics of the system. Input processes arise in many applications, including queuing systems, robotics control with disturbances, and object tracking. Since the state dynamics and rewards depend on the input process, the state alone provides limited information for the expected future returns. Therefore, policy gradient methods with standard state-dependent baselines suffer high variance during training. We derive a bias-free, input-dependent baseline to reduce this variance, and analytically show its benefits over state-dependent baselines. We then propose a meta-learning approach to overcome the complexity of learning a baseline that depends on a long sequence of inputs. Our experimental results show that across environments from queuing systems, computer networks, and MuJoCo robotic locomotion, input-dependent baselines consistently improve training stability and result in better eventual policies.

📄 PDF Abstract BibTeX arXiv:1807.02264

Code (0)

등록된 구현이 없습니다.

Tasks

Meta-LearningMuJoCoObject TrackingPolicy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

On the Reduction of Variance and Overestimation of Deep Q-Learning

2019-10-14 · Mohammed Sabry, Amr M. A. Khalifa

The breakthrough of deep Q-Learning on different types of environments revolutionized the algorithmic design of Reinforcement Learning to introduce more stable and robust algorithms, to that end many extensions to deep Q…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment

2021-07-26 · Jiaming Guo, Rui Zhang, Xishan Zhang, Shaohui Peng 외

Policy gradient methods are appealing in deep reinforcement learning but suffer from high variance of gradient estimate. To reduce the variance, the state value function is applied commonly. However, the effect of the st…

Deep Reinforcement Learning Policy Gradient Methods

Reward Estimation for Variance Reduction in Deep Reinforcement Learning

2018-05-09 · Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet 외

Reinforcement Learning (RL) agents require the specification of a reward signal for learning behaviours. However, introduction of corrupt or stochastic rewards can yield high variance in learning. Such corruption may be …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Variance Reduced Domain Randomization for Policy Gradient

2021-09-29 · Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou 외

By introducing randomness on environment parameters that fundamentally affect the dynamics, domain randomization (DR) imposes diversity to the policy trained by deep reinforcement learning, and thus improves its capabili…