CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Control
Intrinsic motivation is a promising exploration technique for solving reinforcement learning tasks with sparse or absent extrinsic rewards. There exist two technical challenges in implementing intrinsic motivation: 1) how to design a proper intrinsic objective to facilitate efficient exploration; and 2) how to combine the intrinsic objective with the extrinsic objective to help find better solutions. In the current literature, the intrinsic objectives are all designed in a task-agnostic manner and combined with the extrinsic objective via simple addition (or used by itself for reward-free pre-training). In this work, we show that these designs would fail in typical sparse-reward continuous control tasks. To address the problem, we propose Constrained Intrinsic Motivation (CIM) to leverage readily attainable task priors to construct a constrained intrinsic objective, and at the same time, exploit the Lagrangian method to adaptively balance the intrinsic and extrinsic objectives via a simultaneous-maximization framework. We empirically show, on multiple sparse-reward continuous control tasks, that our CIM approach achieves greatly improved performance and sample efficiency over state-of-the-art methods. Moreover, the key techniques of our CIM can also be plugged into existing methods to boost their performances.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlEfficient ExplorationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning
Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+2Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning
Sparsity of rewards while applying a deep reinforcement learning method negatively affects its sample-efficiency. A viable solution to deal with the sparsity of rewards is to learn via intrinsic motivation which advocate…
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Constrained Intrinsic Motivation for Reinforcement Learning
This paper investigates two fundamental problems that arise when utilizing Intrinsic Motivation (IM) for reinforcement learning in Reward-Free Pre-Training (RFPT) tasks and Exploration with Intrinsic Motivation (EIM) tas…
MuJoCoreinforcement-learningReinforcement LearningIntrinsic motivation as constrained entropy maximization
"Intrinsic motivation" refers to the capacity for intelligent systems to be motivated endogenously, i.e. by features of agential architecture itself rather than by learned associations between action and reward. This pap…
Joint Intrinsic Motivation for Coordinated Exploration in Multi-Agent Deep Reinforcement Learning
Multi-agent deep reinforcement learning (MADRL) problems often encounter the challenge of sparse rewards. This challenge becomes even more pronounced when coordination among agents is necessary. As performance depends no…
Deep Reinforcement Learning