GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
Designing agents, capable of learning autonomously a wide range of skills is critical in order to increase the scope of reinforcement learning. It will both increase the diversity of learned skills and reduce the burden of manually designing reward functions for each skill. Self-supervised agents, setting their own goals, and trying to maximize the diversity of those goals have shown great promise towards this end. However, a currently known limitation of agents trying to maximize the diversity of sampled goals is that they tend to get attracted to noise or more generally to parts of the environments that cannot be controlled (distractors). When agents have access to predefined goal features or expert knowledge, absolute Learning Progress (ALP) provides a way to distinguish between regions that can be controlled and those that cannot. However, those methods often fall short when the agents are only provided with raw sensory inputs such as images. In this work we extend those concepts to unsupervised image-based goal exploration. We propose a framework that allows agents to autonomously identify and ignore noisy distracting regions while searching for novelty in the learnable regions to both improve overall performance and avoid catastrophic forgetting. Our framework can be combined with any state-of-the-art novelty seeking goal exploration approaches. We construct a rich 3D image based environment with distractors. Experiments on this environment show that agents using our framework successfully identify interesting regions of the environment, resulting in drastically improved performances. The source code is available at https://sites.google.com/view/grimgep.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningDiversityreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning
Current image-based reinforcement learning (RL) algorithms typically operate on the whole image without performing object-level reasoning. This leads to inefficient goal sampling and ineffective reward functions. In this…
Multi-Goal Reinforcement LearningObjectreinforcement-learningReinforcement Learning+1Visual Reinforcement Learning with Imagined Goals
For an autonomous agent to fulfill a wide range of user-specified goals at test time, it must be able to learn broadly applicable and general-purpose skill repertoires. Furthermore, to provide the requisite level of gene…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Representation LearningGoal-Oriented Visual Question Generation via Intermediate Rewards
Despite significant progress in a variety of vision-and-language problems, developing a method capable of asking intelligent, goal-oriented questions about images is proven to be an inscrutable challenge. Towards this en…
Deep Reinforcement LearningInformativenessQuestion GenerationQuestion-Generation+2Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards
Despite significant progress in a variety of vision-and-language problems, developing a method capable of asking intelligent, goal-oriented questions about images is proven to be an inscrutable challenge. Towards this en…
Deep Reinforcement LearningInformativenessQuestion GenerationQuestion-Generation+2Curriculum goal masking for continuous deep reinforcement learning
Deep reinforcement learning has recently gained a focus on problems where policy or value functions are independent of goals. Evidence exists that the sampling of goals has a strong effect on the learning performance, bu…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)