Principled Exploration via Optimistic Bootstrapping and Backward Induction

2021-05-13 · Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, Zhaoran Wang

One principled approach for provably efficient exploration is incorporating the upper confidence bound (UCB) into the value function as a bonus. However, UCB is specified to deal with linear and tabular settings and is incompatible with Deep Reinforcement Learning (DRL). In this paper, we propose a principled exploration method for DRL through Optimistic Bootstrapping and Backward Induction (OB2I). OB2I constructs a general-purpose UCB-bonus through non-parametric bootstrap in DRL. The UCB-bonus estimates the epistemic uncertainty of state-action pairs for optimistic exploration. We build theoretical connections between the proposed UCB-bonus and the LSVI-UCB in a linear setting. We propagate future uncertainty in a time-consistent manner through episodic backward update, which exploits the theoretical advantage and empirically improves the sample-efficiency. Our experiments in the MNIST maze and Atari suite suggest that OB2I outperforms several state-of-the-art exploration approaches.

📄 PDF Abstract BibTeX arXiv:2105.06022

Code (1)

Baichenjia/OB2I 공식 구현 tf

Tasks

Deep Reinforcement LearningEfficient ExplorationReinforcement Learning (RL)

Optimistic Exploration with Backward Bootstrapped Bonus for Deep Reinforcement Learning

2021-01-01 · Chenjia Bai, Lingxiao Wang, Peng Liu, Zhaoran Wang 외

Optimism in the face of uncertainty is a principled approach for provably efficient exploration for reinforcement learning in tabular and linear settings. However, such an approach is challenging in developing practical …

Atari Games Deep Reinforcement Learning Efficient Exploration Q-Learning+3

Optimistic Exploration even with a Pessimistic Initialisation

2020-02-26 · ICLR 2020 1 · Tabish Rashid, Bei Peng, Wendelin Böhmer, Shimon Whiteson

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algor…

Efficient Exploration Q-Learning Reinforcement Learning Reinforcement Learning (RL)

Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning

2026-02-10 · Akshay Mete, Shahid Aamir Sheikh, Tzu-Hsiang Lin, Dileep Kalathil 외 arxiv

Efficient exploration remains a central challenge in reinforcement learning (RL), particularly in sparse-reward environments. We introduce Optimistic World Models (OWMs), a principled and scalable framework for optimisti…

Reinforcement Learning

Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling

2024-10-07 · Jasmine Bayrooti, Carl Henrik Ek, Amanda Prorok

Learning complex robot behavior through interactions with the environment necessitates principled exploration. Effective strategies should prioritize exploring regions of the state-action space that maximize rewards, wit…

continuous-control Continuous Control Model-based Reinforcement Learning MuJoCo+3

General Exploratory Bonus for Optimistic Exploration in RLHF

2025-09-27 · Wendi Li, Changdae Oh, Sharon Li arxiv

Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide…