paper-with-me

홈 › Papers

Beyond the Boundaries of Proximal Policy Optimization

2024-11-01 · Charlie B. Tan, Edan Toledo, Benjamin Ellis, Jakob N. Foerster, Ferenc Huszár

Proximal policy optimization (PPO) is a widely-used algorithm for on-policy reinforcement learning. This work offers an alternative perspective of PPO, in which it is decomposed into the inner-loop estimation of update vectors, and the outer-loop application of updates using gradient ascent with unity learning rate. Using this insight we propose outer proximal policy optimization (outer-PPO); a framework wherein these update vectors are applied using an arbitrary gradient-based optimizer. The decoupling of update estimation and update application enabled by outer-PPO highlights several implicit design choices in PPO that we challenge through empirical investigation. In particular we consider non-unity learning rates and momentum applied to the outer loop, and a momentum-bias applied to the inner estimation loop. Methods are evaluated against an aggressively tuned PPO baseline on Brax, Jumanji and MinAtar environments; non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget.

📄 PDF Abstract BibTeX arXiv:2411.00666

Code (0)

등록된 구현이 없습니다.

Tasks

Unity

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음
PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

Similar Papers 제목 키워드 기반

Sampling Complexity of TD and PPO in RKHS

2025-09-29 · Lu Zou, Wendi Ren, Weizhong Zhang, Liang Ding 외 arxiv

We revisit Proximal Policy Optimization (PPO) from a function-space perspective. Our analysis decouples policy evaluation and improvement in a reproducing kernel Hilbert space (RKHS): (i) A kernelized temporal-difference…

Stochastic Optimization

Distributed Proximal Policy Optimization for Contention-Based Spectrum Access

2021-10-07 · Akash Doshi, Jeffrey G. Andrews

The increasing number of wireless devices operating in unlicensed spectrum motivates the development of intelligent adaptive approaches to spectrum access that go beyond traditional carrier sensing. We develop a novel di…

Fairness

Riemannian Proximal Policy Optimization

2020-05-19 · Shijun Wang, Baocheng Zhu, Chen Li, Mingzhe Wu 외

In this paper, We propose a general Riemannian proximal optimization algorithm with guaranteed convergence to solve Markov decision process (MDP) problems. To model policy functions in MDP, we employ Gaussian mixture mod…

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

2026-06-16 · Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang 외 arxiv

Knowledge distillation transfers a teacher's competence to a small student but is brittle in the small-student regime: forcing the student to imitate logits from a much larger teacher concentrates it on the teacher's sha…

Knowledge DistillationReinforcement Learning

Proximal Policy Optimization with Mixed Distributed Training

2019-07-15 · Zhen-Yu Zhang, Xiangfeng Luo, Tong Liu, Shaorong Xie 외

Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algori…

Deep Reinforcement LearningReinforcement Learning