Distributional Policy Optimization: An Alternative Approach for Continuous Control
We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric distribution classes. We show that optimizing over such sets results in local movement in the action space and thus convergence to sub-optimal solutions. We suggest a novel distributional framework, able to represent arbitrary distribution functions over the continuous action space. Using this framework, we construct a generative scheme, trained using an off-policy actor-critic paradigm, which we call the Generative Actor Critic (GAC). Compared to policy gradient methods, GAC does not require knowledge of the underlying probability distribution, thereby overcoming these limitations. Empirical evaluation shows that our approach is comparable and often surpasses current state-of-the-art baselines in continuous domains.
Code (3)
Tasks
continuous-controlContinuous ControlPolicy Gradient MethodsSimilar Papers 제목 키워드 기반
Continuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms
Distributed Distributional DrQ is a model-free and off-policy RL algorithm for continuous control tasks based on the state and observation of the agent, which is an actor-critic method with the data-augmentation and the …
continuous-controlContinuous ControlData Augmentationreinforcement-learning+1Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach
Actor-critic algorithms that make use of distributional policy evaluation have frequently been shown to outperform their non-distributional counterparts on many challenging control tasks. Examples of this behavior includ…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning (RL)Distributionally Robust Path Integral Control
We consider a continuous-time continuous-space stochastic optimal control problem, where the controller lacks exact knowledge of the underlying diffusion process, relying instead on a finite set of historical disturbance…
Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the retur…
continuous-controlContinuous ControlDeep Reinforcement LearningDistributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSA…
continuous-controlContinuous ControlMuJoCoreinforcement-learning+2