Proximal Policy Optimization with Continuous Bounded Action Space via the Beta Distribution

2021-11-03 · Irving G. B. Petrazzini, Eric A. Antonelo

Reinforcement learning methods for continuous control tasks have evolved in recent years generating a family of policy gradient methods that rely primarily on a Gaussian distribution for modeling a stochastic policy. However, the Gaussian distribution has an infinite support, whereas real world applications usually have a bounded action space. This dissonance causes an estimation bias that can be eliminated if the Beta distribution is used for the policy instead, as it presents a finite support. In this work, we investigate how this Beta policy performs when it is trained by the Proximal Policy Optimization (PPO) algorithm on two continuous control tasks from OpenAI gym. For both tasks, the Beta policy is superior to the Gaussian policy in terms of agent's final expected reward, also showing more stability and faster convergence of the training process. For the CarRacing environment with high-dimensional image input, the agent's success rate was improved by 63% over the Gaussian policy.

📄 PDF Abstract BibTeX arXiv:2111.02202

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control OpenAI GymPolicy Gradient Methods

Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schrödinger Bridge

2026-03-23 · Yuehu Gong, Zeyuan Wang, Yulin Chen, Shutong Ding 외 arxiv

Classical on-policy algorithms such as PPO and mirror descent policy optimization provide stable proximal policy updates through tractable action likelihoods, but are typically instantiated with simple Gaussian policies …

Reinforcement Learning

PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

2018-10-05 · Perttu Hämäläinen, Amin Babadi, Xiaoxiao Ma, Jaakko Lehtinen

Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which le…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

Proximal Deterministic Policy Gradient

2020-08-03 · Marco Maggipinto, Gian Antonio Susto, Pratik Chaudhari

This paper introduces two simple techniques to improve off-policy Reinforcement Learning (RL) algorithms. First, we formulate off-policy RL as a stochastic proximal point iteration. The target network plays the role of t…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning (RL)

Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)

2023-02-01 · Amartya Mukherjee, Jun Liu

This paper introduces the Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO) algorithm into reinforcement learning. The Hamilton-Jacobi-Bellman (HJB) equation is used in control theory to evaluate the optimali…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sampling Complexity of TD and PPO in RKHS

2025-09-29 · Lu Zou, Wendi Ren, Weizhong Zhang, Liang Ding 외 arxiv

We revisit Proximal Policy Optimization (PPO) from a function-space perspective. Our analysis decouples policy evaluation and improvement in a reproducing kernel Hilbert space (RKHS): (i) A kernelized temporal-difference…

Stochastic Optimization