A Large Deviations Perspective on Policy Gradient Algorithms

2023-11-13 · Wouter Jongeneel, Daniel Kuhn, Mengmeng Li

Motivated by policy gradient methods in the context of reinforcement learning, we identify a large deviation rate function for the iterates generated by stochastic gradient descent for possibly non-convex objectives satisfying a Polyak-{\L}ojasiewicz condition. Leveraging the contraction principle from large deviations theory, we illustrate the potential of this result by showing how convergence properties of policy gradient with a softmax parametrization and an entropy regularized objective can be naturally extended to a wide spectrum of other policy parametrizations.

📄 PDF Abstract BibTeX arXiv:2311.07411

Code (0)

등록된 구현이 없습니다.

Tasks

Policy Gradient Methodsreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Optimal Data Acquisition for Reinforcement Learning: A Large Deviations Perspective

2026-05-27 · Mingjie Hu, Jian-Qiang Hu, Enlu Zhou arxiv

Data acquisition efficiency is a central challenge in deploying reinforcement learning in business and healthcare operations, where interactions are costly, slow, and often involve humans in the loop. This paper develops…

Reinforcement Learning

Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

2022-01-12 · ICLR 2022 4 · Benjamin Bowman, Guido Montufar

We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that in the underparameterized regime the network learns eigenfunctions of an integral operato…

On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient

2010-12-01 · NeurIPS 2010 12 · Tang Jie, Pieter Abbeel

Likelihood ratio policy gradient methods have been some of the most successful reinforcement learning algorithms, especially for learning on physical systems. We describe how the likelihood ratio policy gradient can be d…

Policy Gradient Methods Reinforcement Learning

A weak convergence approach to large deviations for stochastic approximations

2025-02-04 · Henrik Hult, Adam Lindhe, Pierre Nyquist, Guo-Jhen Wu

The theory of stochastic approximations form the theoretical foundation for studying convergence properties of many popular recursive learning algorithms in statistics, machine learning and statistical physics. Large dev…

The Mirage of Action-Dependent Baselines in Reinforcement Learning

2018-02-27 · ICML 2018 7 · George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner 외

Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)