Total stochastic gradient algorithms and applications in reinforcement learning

2019-02-05 · NeurIPS 2018 12 · Paavo Parmas

Backpropagation and the chain rule of derivatives have been prominent; however, the total derivative rule has not enjoyed the same amount of attention. In this work we show how the total derivative rule leads to an intuitive visual framework for creating gradient estimators on graphical models. In particular, previous "policy gradient theorems" are easily derived. We derive new gradient estimators based on density estimation, as well as a likelihood ratio gradient, which "jumps" to an intermediate node, not directly to the objective function. We evaluate our methods on model-based policy gradient algorithms, achieve good performance, and present evidence towards demystifying the success of the popular PILCO algorithm.

📄 PDF Abstract BibTeX arXiv:1902.01722

Code (0)

등록된 구현이 없습니다.

Tasks

Density Estimation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improved Sample Complexity for Stochastic Compositional Variance Reduced Gradient

2018-06-01 · Tianyi Lin, Chenyou Fan, Mengdi Wang, Michael. I. Jordan

Convex composition optimization is an emerging topic that covers a wide range of applications arising from stochastic optimal control, reinforcement learning and multi-stage stochastic programming. Existing algorithms su…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Closing the Gap: Tighter Analysis of Alternating Stochastic Gradient Methods for Bilevel Problems

2021-12-01 · NeurIPS 2021 12 · Tianyi Chen, Yuejiao Sun, Wotao Yin

Stochastic nested optimization, including stochastic compositional, min-max, and bilevel optimization, is gaining popularity in many machine learning applications. While the three problems share a nested structure, exist…

Bilevel Optimization

Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learning

2023-02-22 · Aymeric Dieuleveut, Gersende Fort, Eric Moulines, Hoi-To Wai

Stochastic Approximation (SA) is a classical algorithm that has had since the early days a huge impact on signal processing, and nowadays on machine learning, due to the necessity to deal with a large amount of data obse…

Riemannian Stochastic Gradient Method for Nested Composition Optimization

2022-07-19 · Dewei Zhang, Sam Davanloo Tajbakhsh

This work considers optimization of composition of functions in a nested form over Riemannian manifolds where each function contains an expectation. This type of problems is gaining popularity in applications such as pol…

Meta-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

2022-05-25 · Saeed Masiha, Saber Salehkaleybar, Niao He, Negar Kiyavash 외

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with $1\le\alpha\le2$ which holds in a wide range of applications in machine learning …

Policy Gradient Methods Reinforcement Learning (RL)Second-order methods