Proximal Policy Optimization Smoothed Algorithm

2020-12-04 · Wangshu Zhu, Andre Rosendo

Proximal policy optimization (PPO) has yielded state-of-the-art results in policy search, a subfield of reinforcement learning, with one of its key points being the use of a surrogate objective function to restrict the step size at each policy update. Although such restriction is helpful, the algorithm still suffers from performance instability and optimization inefficiency from the sudden flattening of the curve. To address this issue we present a PPO variant, named Proximal Policy Optimization Smooth Algorithm (PPOS), and its critical improvement is the use of a functional clipping method instead of a flat clipping method. We compare our method with PPO and PPORB, which adopts a rollback clipping method, and prove that our method can conduct more accurate updates at each time step than other PPO methods. Moreover, we show that it outperforms the latest PPO variants on both performance and stability in challenging continuous control tasks.

📄 PDF Abstract BibTeX arXiv:2012.02439

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Smoothed Action Value Functions for Learning Gaussian Policies

2018-03-06 · ICML 2018 7 · Ofir Nachum, Mohammad Norouzi, George Tucker, Dale Schuurmans

State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian s…

continuous-control Continuous Control Q-Learning Reinforcement Learning+1

Learning Gaussian Policies from Smoothed Action Value Functions

2018-01-01 · ICLR 2018 1 · Ofir Nachum, Mohammad Norouzi, George Tucker, Dale Schuurmans

continuous-control Continuous Control Q-Learning Reinforcement Learning+1

Smooth, Sparse, and Stable: Finite-Time Exact Skeleton Recovery via Smoothed Proximal Gradients

2026-01-26 · Rui Wu, Yongjun Li arxiv

Continuous optimization has significantly advanced causal discovery, yet existing methods (e.g., NOTEARS) generally guarantee only asymptotic convergence to a stationary point. This often yields dense weighted matrices t…

ESPO: Early-Stopping Proximal Policy Optimization

2026-05-28 · Zihang Li, Rui Zhou, Yingcheng Shi, Wenhan Yu 외 arxiv

When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that ne…

Reinforcement Learning Mathematical Reasoning

Policy Optimization in Zero-Sum Markov Games: Fictitious Self-Play Provably Attains Nash Equilibria

2021-01-01 · Boyi Liu, Zhuoran Yang, Zhaoran Wang

Fictitious Self-Play (FSP) has achieved significant empirical success in solving extensive-form games. However, from a theoretical perspective, it remains unknown whether FSP is guaranteed to converge to Nash equilibria…