Structure Matters: Dynamic Policy Gradient

2024-11-07 · Sara Klein, Xiangyuan Zhang, Tamer Başar, Simon Weissmann, Leif Döring

In this work, we study $\gamma$-discounted infinite-horizon tabular Markov decision processes (MDPs) and introduce a framework called dynamic policy gradient (DynPG). The framework directly integrates dynamic programming with (any) policy gradient method, explicitly leveraging the Markovian property of the environment. DynPG dynamically adjusts the problem horizon during training, decomposing the original infinite-horizon MDP into a sequence of contextual bandit problems. By iteratively solving these contextual bandits, DynPG converges to the stationary optimal policy of the infinite-horizon MDP. To demonstrate the power of DynPG, we establish its non-asymptotic global convergence rate under the tabular softmax parametrization, focusing on the dependencies on salient but essential parameters of the MDP. By combining classical arguments from dynamic programming with more recent convergence arguments of policy gradient schemes, we prove that softmax DynPG scales polynomially in the effective horizon $(1-\gamma)^{-1}$. Our findings contrast recent exponential lower bound examples for vanilla policy gradient.

📄 PDF Abstract BibTeX arXiv:2411.04913

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-Armed Bandits

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Gradient Information Matters in Policy Optimization by Back-propagating through Model

2021-09-29 · ICLR 2022 4 · Chongchong Li, Yue Wang, Wei Chen, YuTing Liu 외

Model-based reinforcement learning provides an efficient mechanism to find the optimal policy by interacting with the learned environment. In addition to treating the learned environment like a black-box simulator, a mor…

continuous-control Continuous Control Model-based Reinforcement Learning

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

2020-05-25 · Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras 외

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, …

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

2023-10-04 · Sara Klein, Simon Weissmann, Leif Döring

Markov Decision Processes (MDPs) are a formal framework for modeling and solving sequential decision-making problems. In finite-time horizons such problems are relevant for instance for optimal stopping or specific suppl…

Decision Making Policy Gradient Methods Sequential Decision Making

Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL

2019-04-08 · Yannis Flet-Berliac, Philippe Preux

In reinforcement learning, policy gradient algorithms optimize the policy directly and rely on sampling efficiently an environment. Nevertheless, while most sampling procedures are based on direct policy sampling, self-p…

continuous-control Continuous Control Denoising Policy Gradient Methods+1

Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning

2025-07-30 · Afshin Khadangi, Amir Sartipi, Igor Tchappi, Ramin Bahmani 외 arxiv

The tension between data privacy and model utility has become the defining bottleneck for the practical deployment of large language models (LLMs) trained on sensitive corpora including healthcare. Differentially private…

Reinforcement Learning