Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Models

2024-10-09 · Alain Andres, Javier Del Ser

Exploration remains a significant challenge in reinforcement learning, especially in environments where extrinsic rewards are sparse or non-existent. The recent rise of foundation models, such as CLIP, offers an opportunity to leverage pretrained, semantically rich embeddings that encapsulate broad and reusable knowledge. In this work we explore the potential of these foundation models not just to drive exploration, but also to analyze the critical role of the episodic novelty term in enhancing exploration effectiveness of the agent. We also investigate whether providing the intrinsic module with complete state information -- rather than just partial observations -- can improve exploration, despite the difficulties in handling small variations within large state spaces. Our experiments in the MiniGrid domain reveal that intrinsic modules can effectively utilize full state information, significantly increasing sample efficiency while learning an optimal policy. Moreover, we show that the embeddings provided by foundation models are sometimes even better than those constructed by the agent during training, further accelerating the learning process, especially when coupled with the episodic novelty term to enhance exploration.

📄 PDF Abstract BibTeX arXiv:2410.07404

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

Intrinsic motivation in virtual assistant interaction for fostering spontaneous interactions

2020-10-13 · Chang Li, Hideyoshi Yanagisawa

With the growing utility of today's conversational virtual assistants, the importance of user motivation in human-AI interaction is becoming more obvious. However, previous studies in this and related fields, such as hum…

Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop

2018-06-21 · Martin Biehl, Christian Guckelsberger, Christoph Salge, Simón C. Smith 외

Active inference is an ambitious theory that treats perception, inference and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognit…

Reinforcement Learning

Efficient Exploration through Intrinsic Motivation Learning for Unsupervised Subgoal Discovery in Model-Free Hierarchical Reinforcement Learning

2019-11-18 · Jacob Rafati, David C. Noelle

Efficient exploration for automatic subgoal discovery is a challenging problem in Hierarchical Reinforcement Learning (HRL). In this paper, we show that intrinsic motivation learning increases the efficiency of explorati…

Efficient Exploration Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review

2023-01-24 · Artem Latyshev, Aleksandr I. Panov

The reinforcement learning research area contains a wide range of methods for solving the problems of intelligent agent control. Despite the progress that has been made, the task of creating a highly autonomous agent is …

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

PreND: Enhancing Intrinsic Motivation in Reinforcement Learning through Pre-trained Network Distillation

2024-10-02 · Mohammadamin Davoodabadi, Negin Hashemi Dijujin, Mahdieh Soleymani Baghshah

Intrinsic motivation, inspired by the psychology of developmental learning in infants, stimulates exploration in agents without relying solely on sparse external rewards. Existing methods in reinforcement learning like R…

Developmental Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)