SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation

2017-12-29 · ICML 2018 7 · Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, Le Song

When function approximation is used, solving the Bellman optimality equation with stability guarantees has remained a major open problem in reinforcement learning for decades. The fundamental difficulty is that the Bellman operator may become an expansion in general, resulting in oscillating and even divergent behavior of popular algorithms like Q-learning. In this paper, we revisit the Bellman equation, and reformulate it into a novel primal-dual optimization problem using Nesterov's smoothing technique and the Legendre-Fenchel transformation. We then develop a new algorithm, called Smoothed Bellman Error Embedding, to solve this optimization problem where any differentiable function class may be used. We provide what we believe to be the first convergence guarantee for general nonlinear function approximation, and analyze the algorithm's sample complexity. Empirically, our algorithm compares favorably to state-of-the-art baselines in several benchmark control problems.

📄 PDF Abstract BibTeX arXiv:1712.10285

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sharp Analysis of Smoothed Bellman Error Embedding

2020-07-07 · Ahmed Touati, Pascal Vincent

The \textit{Smoothed Bellman Error Embedding} algorithm~\citep{dai2018sbeed}, known as SBEED, was proposed as a provably convergent reinforcement learning algorithm with general nonlinear function approximation. It has b…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Convergent Reinforcement Learning with Function Approximation: A Bilevel Optimization Perspective

2018-09-27 · Zhuoran Yang, Zuyue Fu, Kaiqing Zhang, Zhaoran Wang

We study reinforcement learning algorithms with nonlinear function approximation in the online setting. By formulating both the problems of value function estimation and policy learning as bilevel optimization problems,…

Bilevel Optimization Q-Learning reinforcement-learning Reinforcement Learning+1

Convergent Policy Optimization for Safe Reinforcement Learning

2019-10-26 · NeurIPS 2019 12 · Ming Yu, Zhuoran Yang, Mladen Kolar, Zhaoran Wang

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonco…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation

2019-11-11 · ICML 2020 1 · Shangtong Zhang, Bo Liu, Hengshuai Yao, Shimon Whiteson

We present the first provably convergent two-timescale off-policy actor-critic algorithm (COF-PAC) with function approximation. Key to COF-PAC is the introduction of a new critic, the emphasis critic, which is trained vi…

Vocal Bursts Valence Prediction

Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

2025-08-19 · Soumyajit Guin, Shalabh Bhatnagar arxiv

In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in R…

Reinforcement Learning