Sharp Analysis of Smoothed Bellman Error Embedding

2020-07-07 · Ahmed Touati, Pascal Vincent

The \textit{Smoothed Bellman Error Embedding} algorithm~\citep{dai2018sbeed}, known as SBEED, was proposed as a provably convergent reinforcement learning algorithm with general nonlinear function approximation. It has been successfully implemented with neural networks and achieved strong empirical results. In this work, we study the theoretical behavior of SBEED in batch-mode reinforcement learning. We prove a near-optimal performance guarantee that depends on the representation power of the used function classes and a tight notion of the distribution shift. Our results improve upon prior guarantees for SBEED in ~\citet{dai2018sbeed} in terms of the dependence on the planning horizon and on the sample size. Our analysis builds on the recent work of ~\citet{Xie2020} which studies a related algorithm MSBO, that could be interpreted as a \textit{non-smooth} counterpart of SBEED.

📄 PDF Abstract BibTeX arXiv:2007.03749

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation

2017-12-29 · ICML 2018 7 · Bo Dai, Albert Shaw, Lihong Li, Lin Xiao 외

When function approximation is used, solving the Bellman optimality equation with stability guarantees has remained a major open problem in reinforcement learning for decades. The fundamental difficulty is that the Bellm…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework

2026-05-11 · Phalguni Nanda, Zaiwei Chen arxiv

In this work, we show that natural policy gradient, a core algorithm in reinforcement learning, admits an exact formulation as a smoothed and averaged form of policy iteration. Specifically, we introduce doubly smoothed …

Reinforcement Learning

Learning the Supports for Categorical Critic in Reinforcement Learning

2026-07-02 · Jen-Yen Chang, Takayuki Osa, Tatsuya Harada arxiv

Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to b…

Reinforcement Learning

Structured Linear Contextual Bandits: A Sharp and Geometric Smoothed Analysis

2020-02-26 · ICML 2020 1 · Vidyashankar Sivakumar, Zhiwei Steven Wu, Arindam Banerjee

Bandit learning algorithms typically involve the balance of exploration and exploitation. However, in many practical applications, worst-case scenarios needing systematic exploration are seldom encountered. In this work,…

Multi-Armed Bandits

Solving Dynamic Discrete Choice Models Using Smoothing and Sieve Methods

2020-02-28

We propose to combine smoothing, simulations and sieve approximations to solve for either the integrated or expected value function in a general class of dynamic discrete choice (DDC) models. We use importance sampling t…

Discrete Choice Models Gaussian Processes