Learning to Plan via Deep Optimistic Value Exploration

2020-06-08 · L4DC 2020 6 · Tim Seyde, Wilko Schwarting, Sertac Karaman, Daniela Rus

Deep exploration requires coordinated long-term planning. We present a model-based reinforcement learning algorithm that guides policy learning through a value function that exhibits optimism in the face of uncertainty. We capture uncertainty over values by combining predictions from an ensemble of models and formulate an upper confidence bound (UCB) objective to recover optimistic estimates. Training the policy on ensemble rollouts with the learned value function as the terminal cost allows for projecting long-term interactions into a limited planning horizon, thus enabling deep optimistic exploration. We do not assume a priori knowledge of either the dynamics or reward function. We demonstrate that our approach can accommodate both dense and sparse reward signals, while improving sample complexity on a variety of benchmarking tasks.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

BenchmarkingModel-based Reinforcement LearningReinforcement Learning (RL)

Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles

2020-10-27 · Tim Seyde, Wilko Schwarting, Sertac Karaman, Daniela Rus

Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conduc…

Decision Making

Optimistic Exploration even with a Pessimistic Initialisation

2020-02-26 · ICLR 2020 1 · Tabish Rashid, Bei Peng, Wendelin Böhmer, Shimon Whiteson

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algor…

Efficient Exploration Q-Learning Reinforcement Learning Reinforcement Learning (RL)

DOP: Deep Optimistic Planning with Approximate Value Function Evaluation

2018-03-22 · Francesco Riccio, Roberto Capobianco, Daniele Nardi

Research on reinforcement learning has demonstrated promising results in manifold applications and domains. Still, efficiently learning effective robot behaviors is very difficult, due to unstructured scenarios, high unc…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Randomized Exploration for Reinforcement Learning with General Value Function Approximation

2021-06-15 · Haque Ishfaq, Qiwen Cui, Viet Nguyen, Alex Ayoub 외

We propose a model-free reinforcement learning algorithm inspired by the popular randomized least squares value iteration (RLSVI) algorithm as well as the optimism principle. Unlike existing upper-confidence-bound (UCB) …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning

2025-02-05 · Ruoning Zhang, Siying Wang, Wenyu Chen, Yang Zhou 외

The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, due to the representational limitations of traditional monotonic value deco…

Multi-agent Reinforcement Learning