Policy Optimization with Second-Order Advantage Information
Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework to reduce the variance. To invoke RB, our proposed algorithm (POSA) learns the underlying factorization structure among the action space based on the second-order advantage information. POSA captures the quadratic information explicitly and efficiently by utilizing the wide & deep architecture. Empirical studies show that our proposed approach demonstrates the performance improvements on high-dimensional synthetic settings and OpenAI Gym's MuJoCo continuous control tasks.
Code (1)
Tasks
continuous-controlContinuous ControlMuJoCoSimilar Papers 제목 키워드 기반
Momentum-Based Policy Gradient with Second-Order Information
Variance-reduced gradient estimators for policy gradient methods have been one of the main focus of research in the reinforcement learning in recent years as they allow acceleration of the estimation process. We propose …
Policy Gradient MethodsPolicy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
As the most successful variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely applied across various domains with several advantages: efficient data utiliz…
Atari GamesReinforcement LearningStochastic Dimension-reduced Second-order Methods for Policy Optimization
In this paper, we propose several new stochastic second-order algorithms for policy optimization that only require gradient and Hessian-vector product in each iteration, making them computationally efficient and comparab…
Policy Gradient MethodsSecond-order methodsCooperative Multi-Agent Reinforcement Learning with Partial Observations
In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of …
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
We address the discounted reward setting in reinforcement learning (RL). To mitigate the value approximation challenges in policy gradient methods, actor-critic approaches have been developed and are known to converge to…
Reinforcement Learning