Policy Optimization with Stochastic Mirror Descent

2019-06-25 · Long Yang, Yu Zhang, Gang Zheng, Qian Zheng, Pengfei Li, Jianhang Huang, Jun Wen, Gang Pan

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes $\mathtt{VRMPO}$ algorithm: a sample efficient policy gradient method with stochastic mirror descent. In $\mathtt{VRMPO}$, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency. We prove that the proposed $\mathtt{VRMPO}$ needs only $\mathcal{O}(\epsilon^{-3})$ sample trajectories to achieve an $\epsilon$-approximate first-order stationary point, which matches the best sample complexity for policy optimization. The extensive experimental results demonstrate that $\mathtt{VRMPO}$ outperforms the state-of-the-art policy gradient methods in various settings.

📄 PDF Abstract BibTeX arXiv:1906.10462

Code (0)

등록된 구현이 없습니다.

Tasks

Continuous ControlPolicy Gradient Methodsreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Mirror Descent on Riemannian Manifolds

2026-03-18 · Jiaxin Jiang, Lei Shi, Jiyuan Tan arxiv

Mirror Descent (MD) is a scalable first-order method widely used in large-scale optimization, with applications in image processing, policy optimization, and neural network training. This paper generalizes MD to optimiza…

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

2026-03-18 · Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen 외 arxiv

Balancing policy expressiveness with the exploration-exploitation trade-off is a core challenge in online Reinforcement Learning (RL). While Stochastic Differential Equation (SDE)-based diffusion policies can represent c…

Reinforcement Learning

Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrors

2020-09-18 · Adityanarayanan Radhakrishnan, Mikhail Belkin, Caroline Uhler

The Polyak-Lojasiewicz (PL) inequality is a sufficient condition for establishing linear convergence of gradient descent, even in non-convex settings. While several recent works use a PL-based analysis to establish linea…

Value Mirror Descent for Reinforcement Learning

2026-04-07 · Zhichao Jia, Guanghui Lan arxiv

Value iteration-type methods have been extensively studied for computing a nearly optimal value function in reinforcement learning (RL). Under a generative sampling model, these methods can achieve sharper sample complex…

Reinforcement Learning

Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize

2021-10-28 · Ryan D'Orazio, Nicolas Loizou, Issam Laradji, Ioannis Mitliagkas

We investigate the convergence of stochastic mirror descent (SMD) under interpolation in relatively smooth and smooth convex optimization. In relatively smooth convex optimization we provide new convergence guarantees fo…