Empowerment-driven Exploration using Mutual Information Estimation

2018-10-11 · Navneet Madhu Kumar

Exploration is a difficult challenge in reinforcement learning and is of prime importance in sparse reward environments. However, many of the state of the art deep reinforcement learning algorithms, that rely on epsilon-greedy, fail on these environments. In such cases, empowerment can serve as an intrinsic reward signal to enable the agent to maximize the influence it has over the near future. We formulate empowerment as the channel capacity between states and actions and is calculated by estimating the mutual information between the actions and the following states. The mutual information is estimated using Mutual Information Neural Estimator and a forward dynamics model. We demonstrate that an empowerment driven agent is able to improve significantly the score of a baseline DQN agent on the game of Montezuma's Revenge.

📄 PDF Abstract BibTeX arXiv:1810.05533

Code (1)

navneet-nmk/pytorch-rl 공식 구현 pytorch

Tasks

Deep Reinforcement LearningMontezuma's RevengeMutual Information Estimationreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Unsupervised Skill Discovery through Skill Regions Differentiation

2025-06-17 · Ting Xiao, Jiakun Zheng, Rushuai Yang, Kang Xu 외

Unsupervised Reinforcement Learning (RL) aims to discover diverse behaviors that can accelerate the learning of downstream tasks. Previous methods typically focus on entropy-based exploration or empowerment-driven skill …

Density Estimation Reinforcement Learning (RL)Unsupervised Reinforcement Learning

Towards Empowerment Gain through Causal Structure Learning in Model-Based RL

2025-02-14 · Hongye Cao, Fan Feng, Meng Fang, Shaokang Dong 외

In Model-Based Reinforcement Learning (MBRL), incorporating causal structures into dynamics models provides agents with a structured understanding of the environments, enabling efficient decision. Empowerment as an intri…

Causal Discovery Model-based Reinforcement Learning

Efficient Empowerment Estimation for Unsupervised Stabilization

2020-07-14 · ICLR 2021 1 · Ruihan Zhao, Kevin Lu, Pieter Abbeel, Stas Tiomkin

Intrinsically motivated artificial agents learn advantageous behavior without externally-provided rewards. Previously, it was shown that maximizing mutual information between agent actuators and future states, known as t…

INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL

2022-04-18 · ICLR 2022 4 · Homanga Bharadhwaj, Mohammad Babaeizadeh, Dumitru Erhan, Sergey Levine

Model-based reinforcement learning (RL) algorithms designed for handling complex visual observations typically learn some sort of latent state representation, either explicitly or implicitly. Standard methods of this sor…

Model-based Reinforcement Learning Reinforcement Learning (RL)

Learning Efficient Representation for Intrinsic Motivation

2019-12-04 · Ruihan Zhao, Stas Tiomkin, Pieter Abbeel

Mutual Information between agent Actions and environment States (MIAS) quantifies the influence of agent on its environment. Recently, it was found that the maximization of MIAS can be used as an intrinsic motivation for…

Reinforcement Learning