Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization

2018-07-02 · Xiangxiang Chu

As the most successful variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely applied across various domains with several advantages: efficient data utilization, easy implementation, and good parallelism. In this paper, a first-order gradient reinforcement learning algorithm called Policy Optimization with Penalized Point Probability Distance (POP3D), which is a lower bound to the square of total variance divergence is proposed as another powerful variant. Firstly, we talk about the shortcomings of several commonly used algorithms, by which our method is partly motivated. Secondly, we address to overcome these shortcomings by applying POP3D. Thirdly, we dive into its mechanism from the perspective of solution manifold. Finally, we make quantitative comparisons among several state-of-the-art algorithms based on common benchmarks. Simulation results show that POP3D is highly competitive compared with PPO. Besides, our code is released in https://github.com/paperwithcode/pop3d.

📄 PDF Abstract BibTeX arXiv:1807.00442

Code (2)

cxxgtxy/POP3D 공식 구현 tf

paperwithcode/pop3d 공식 구현 tf

Tasks

Atari Games Reinforcement Learning

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

A Strong On-Policy Competitor To PPO

2021-01-01 · Xiangxiang Chu

As a recognized variant and improvement for Trust Region Policy Optimization (TRPO), proximal policy optimization (PPO) has been widely used with several advantages: efficient data utilization, easy implementation and go…

Modifying Final Splits of Classification Tree for Fine-tuning Subpopulation Target in Policy Making

2025-02-20 · Lei Bill Wang, Zhenbang Jiao, Fangyi Wang

Policymakers often use Classification and Regression Trees (CART) to partition populations based on binary outcomes and target subpopulations whose probability of the binary event exceeds a threshold. However, classic CA…

Knowledge Distillation

Penalized Langevin dynamics with vanishing penalty for smooth and log-concave targets

2020-06-24 · NeurIPS 2020 12 · Avetik Karagulyan, Arnak S. Dalalyan

We study the problem of sampling from a probability distribution on $\mathbb R^p$ defined via a convex and smooth potential function. We consider a continuous-time diffusion-type process, termed Penalized Langevin dynami…

Rethinking the Trust Region in LLM Reinforcement Learning

2026-02-04 · Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang 외 arxiv

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that t…

Reinforcement Learning

Probabilistic hypergraph grammars for efficient molecular optimization

2019-06-05 · Egor Kraev, Mark Harley

We present an approach to make molecular optimization more efficient. We infer a hypergraph replacement grammar from the ChEMBL database, count the frequencies of particular rules being used to expand particular nontermi…

GPU reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)