Optimistic Exploration even with a Pessimistic Initialisation

2020-02-26 · ICLR 2020 1 · Tabish Rashid, Bei Peng, Wendelin Böhmer, Shimon Whiteson

Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient tabular algorithms. In particular, in scenarios with only positive rewards, Q-values are initialised at their lowest possible values due to commonly used network initialisation schemes, a pessimistic initialisation. Merely initialising the network to output optimistic Q-values is not enough, since we cannot ensure that they remain optimistic for novel state-action pairs, which is crucial for exploration. We propose a simple count-based augmentation to pessimistically initialised Q-values that separates the source of optimism from the neural network. We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting. Our algorithm, Optimistic Pessimistically Initialised Q-Learning (OPIQ), augments the Q-value estimates of a DQN-based agent with count-derived bonuses to ensure optimism during both action selection and bootstrapping. We show that OPIQ outperforms non-optimistic DQN variants that utilise a pseudocount-based intrinsic motivation in hard exploration tasks, and that it predicts optimistic estimates for novel state-action pairs.

📄 PDF Abstract BibTeX arXiv:2002.12174

Code (1)

oxwhirl/opiq 공식 구현 pytorch

Tasks

Efficient ExplorationQ-LearningReinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Efficient Reinforcement Learning via Decoupling Exploration and Utilization

2023-12-26 · Jingpu Yang, Helin Wang, Qirui Zhao, Zhecheng Shi 외

Reinforcement Learning (RL), recognized as an efficient learning approach, has achieved remarkable success across multiple fields and applications, including gaming, robotics, and autonomous vehicles. Classical single-ag…

Autonomous Vehicles MuJoCo reinforcement-learning Reinforcement Learning+1

Wasserstein Barycenter Soft Actor-Critic

2025-06-11 · Zahra Shahrooei, Ali Baheri

Deep off-policy actor-critic algorithms have emerged as the leading framework for reinforcement learning in continuous control domains. However, most of these algorithms suffer from poor sample efficiency, especially in …

continuous-control Continuous Control MuJoCo

DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning

2021-12-01 · Archana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai 외

Safe reinforcement learning is extremely challenging--not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations. We formulate this safe reinforcement learning (R…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration+1

Better Exploration with Optimistic Actor-Critic

2019-10-28 · Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of …

continuous-control Continuous Control Efficient Exploration Reinforcement Learning

Better Exploration with Optimistic Actor Critic

2019-12-01 · NeurIPS 2019 12 · Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann