Learning Efficient Representation for Intrinsic Motivation
Mutual Information between agent Actions and environment States (MIAS) quantifies the influence of agent on its environment. Recently, it was found that the maximization of MIAS can be used as an intrinsic motivation for artificial agents. In literature, the term empowerment is used to represent the maximum of MIAS at a certain state. While empowerment has been shown to solve a broad range of reinforcement learning problems, its calculation in arbitrary dynamics is a challenging problem because it relies on the estimation of mutual information. Existing approaches, which rely on sampling, are limited to low dimensional spaces, because high-confidence distribution-free lower bounds for mutual information require exponential number of samples. In this work, we develop a novel approach for the estimation of empowerment in unknown dynamics from visual observation only, without the need to sample for MIAS. The core idea is to represent the relation between action sequences and future states using a stochastic dynamic model in latent space with a specific form. This allows us to efficiently compute empowerment with the "Water-Filling" algorithm from information theory. We construct this embedding with deep neural networks trained on a sophisticated objective function. Our experimental results show that the designed embedding preserves information-theoretic properties of the original dynamics.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Toward Explainable Offline RL: Analyzing Representations in Intrinsically Motivated Decision Transformers
Elastic Decision Transformers (EDTs) have proved to be particularly successful in offline reinforcement learning, offering a flexible framework that unifies sequence modeling with decision-making under uncertainty. Recen…
Decision MakingDecision Making Under UncertaintyOffline RLPreND: Enhancing Intrinsic Motivation in Reinforcement Learning through Pre-trained Network Distillation
Intrinsic motivation, inspired by the psychology of developmental learning in infants, stimulates exploration in agents without relying solely on sparse external rewards. Existing methods in reinforcement learning like R…
Developmental Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning
Efficient exploration for automatic subgoal discovery is a challenging problem in Hierarchical Reinforcement Learning (HRL). In this paper, we show that intrinsic motivation learning increases the efficiency of explorati…
Efficient ExplorationHierarchical Reinforcement Learningreinforcement-learningReinforcement Learning+1Intrinsic motivation in virtual assistant interaction for fostering spontaneous interactions
With the growing utility of today's conversational virtual assistants, the importance of user motivation in human-AI interaction is becoming more obvious. However, previous studies in this and related fields, such as hum…
The intrinsic motivation of reinforcement and imitation learning for sequential tasks
This work in the field of developmental cognitive robotics aims to devise a new domain bridging between reinforcement learning and imitation learning, with a model of the intrinsic motivation for learning agents to learn…
Imitation LearningMulti-Task Learning