Explicit Planning for Efficient Exploration in Reinforcement Learning

2019-12-01 · NeurIPS 2019 12 · Liangpeng Zhang, Ke Tang, Xin Yao

Efficient exploration is crucial to achieving good performance in reinforcement learning. Existing systematic exploration strategies (R-MAX, MBIE, UCRL, etc.), despite being promising theoretically, are essentially greedy strategies that follow some predefined heuristics. When the heuristics do not match the dynamics of Markov decision processes (MDPs) well, an excessive amount of time can be wasted in travelling through already-explored states, lowering the overall efficiency. We argue that explicit planning for exploration can help alleviate such a problem, and propose a Value Iteration for Exploration Cost (VIEC) algorithm which computes the optimal exploration scheme by solving an augmented MDP. We then present a detailed analysis of the exploration behaviour of some popular strategies, showing how these strategies can fail and spend O(n^2 md) or O(n^2 m + nmd) steps to collect sufficient data in some tower-shaped MDPs, while the optimal exploration scheme, which can be obtained by VIEC, only needs O(nmd), where n, m are the numbers of states and actions and d is the data demand. The analysis not only points out the weakness of existing heuristic-based strategies, but also suggests a remarkable potential in explicit planning for exploration.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Efficient Explorationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improving width-based planning with compact policies

2018-06-15 · Miquel Junyent, Anders Jonsson, Vicenç Gómez

Optimal action selection in decision problems characterized by sparse, delayed rewards is still an open challenge. For these problems, current deep reinforcement learning methods require enormous amounts of data to learn…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing

2026-01-14 · Qian Cao, Yahui Liu, Wei Bi, Yi Zhao 외 arxiv

Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit me…

Reinforcement Learning

PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning

2020-04-24 · Guillaume Matheron, Nicolas Perrin, Olivier Sigaud

The exploration-exploitation trade-off is at the heart of reinforcement learning (RL). However, most continuous control benchmarks used in recent RL research only require local exploration. This led to the development of…

continuous-control Continuous Control Efficient Exploration Motion Planning+3

UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

2026-06-17 · Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang, Nicholas Rhinehart arxiv

Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collectio…

Reinforcement Learning

Explicit Explore-Exploit Algorithms in Continuous State Spaces

2019-11-01 · NeurIPS 2019 12 · Mikael Henaff

We present a new model-based algorithm for reinforcement learning (RL) which consists of explicit exploration and exploitation phases, and is applicable in large or infinite state spaces. The algorithm maintains a set of…

Reinforcement Learning Reinforcement Learning (RL)