paper-with-me

홈 › Papers

Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

2026-03-10 · Ji Gao, Caleb Ju, Guanghui Lan, Zhaohui Tong arxiv

Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose \textit{actor-accelerated PDA}, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.

📄 PDF Abstract BibTeX arXiv:2603.10199

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Similar Papers 제목 키워드 기반

Accelerated Dual-Averaging Primal-Dual Method for Composite Convex Minimization

2020-01-15 · Conghui Tan, Yuqiu Qian, Shiqian Ma, Tong Zhang

Dual averaging-type methods are widely used in industrial machine learning applications due to their ability to promoting solution structure (e.g., sparsity) efficiently. In this paper, we propose a novel accelerated dua…

BIG-bench Machine Learning

Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning

2018-02-19 · Qingkai Liang, Fanyu Que, Eytan Modiano

Constrained Markov Decision Process (CMDP) is a natural framework for reinforcement learning tasks with safety constraints, where agents learn a policy that maximizes the long-term reward while satisfying the constraints…

reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning

Adaptive Averaging in Accelerated Descent Dynamics

2016-12-01 · NeurIPS 2016 12 · Walid Krichene, Alexandre Bayen, Peter L. Bartlett

We study accelerated descent dynamics for constrained convex optimization. This dynamics can be described naturally as a coupling of a dual variable accumulating gradients at a given rate $\eta(t)$, and a primal variable…

Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization

2018-06-03 · NeurIPS 2018 12 · Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, Mingyi Hong

Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents. Motivated by decentralized applications such as sensor…

Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)

Accelerated Reinforcement Learning

2017-10-23 · K. Lakshmanan

Policy gradient methods are widely used in reinforcement learning algorithms to search for better policies in the parameterized policy space. They do gradient search in the policy space and are known to converge very slo…

Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2