Multi-Path Policy Optimization

2019-11-11 · Ling Pan, Qingpeng Cai, Longbo Huang

Recent years have witnessed a tremendous improvement of deep reinforcement learning. However, a challenging problem is that an agent may suffer from inefficient exploration, particularly for on-policy methods. Previous exploration methods either rely on complex structure to estimate the novelty of states, or incur sensitive hyper-parameters causing instability. We propose an efficient exploration method, Multi-Path Policy Optimization (MPPO), which does not incur high computation cost and ensures stability. MPPO maintains an efficient mechanism that effectively utilizes a population of diverse policies to enable better exploration, especially in sparse environments. We also give a theoretical guarantee of the stable performance. We build our scheme upon two widely-adopted on-policy methods, the Trust-Region Policy Optimization algorithm and Proximal Policy Optimization algorithm. We conduct extensive experiments on several MuJoCo tasks and their sparsified variants to fairly evaluate the proposed method. Results show that MPPO significantly outperforms state-of-the-art exploration methods in terms of both sample efficiency and final performance.

📄 PDF Abstract BibTeX arXiv:1911.04207

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningEfficient ExplorationMuJoCoReinforcement Learning

Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schrödinger Bridge

2026-03-23 · Yuehu Gong, Zeyuan Wang, Yulin Chen, Shutong Ding 외 arxiv

Classical on-policy algorithms such as PPO and mirror descent policy optimization provide stable proximal policy updates through tractable action likelihoods, but are typically instantiated with simple Gaussian policies …

Reinforcement Learning

Policy path programming

2019-09-25 · Daniel McNamee

We develop a normative theory of hierarchical model-based policy optimization for Markov decision processes resulting in a full-depth, full-width policy iteration algorithm. This method performs policy updates which inte…

All

Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

2017-07-06 · ICLR 2018 1 · Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans

Trust region methods, such as TRPO, are often used to stabilize policy optimization algorithms in reinforcement learning (RL). While current trust region strategies are effective for continuous control, they typically re…

continuous-control Continuous Control Reinforcement Learning Reinforcement Learning (RL)

RL4health: Crowdsourcing Reinforcement Learning for Knee Replacement Pathway Optimization

2019-05-24 · Hao Lu, Mengdi Wang

Joint replacement is the most common inpatient surgical treatment in the US. We investigate the clinical pathway optimization for knee replacement, which is a sequential decision process from onset to recovery. Based on …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Relative Entropy Pathwise Policy Optimization

2025-07-15 · Claas Voelcker, Axel Brunnbauer, Marcel Hussing, Michal Nauman 외

Score-function policy gradients have delivered strong results in game-playing, robotics and language-model fine-tuning. Yet its high-variance often undermines training stability. On the other hand, pathwise policy gradie…

GPU