Skill-Based Reinforcement Learning with Intrinsic Reward Matching

2022-10-14 · Ademi Adeniji, Amber Xie, Pieter Abbeel

While unsupervised skill discovery has shown promise in autonomously acquiring behavioral primitives, there is still a large methodological disconnect between task-agnostic skill pretraining and downstream, task-aware finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two phases of learning via the $\textit{skill discriminator}$, a pretraining model component often discarded during finetuning. Conventional approaches finetune pretrained agents directly at the policy level, often relying on expensive environment rollouts to empirically determine the optimal skill. However, often the most concise yet complete description of a task is the reward function itself, and skill learning methods learn an $\textit{intrinsic}$ reward function via the discriminator that corresponds to the skill policy. We propose to leverage the skill discriminator to $\textit{match}$ the intrinsic and downstream task rewards and determine the optimal skill for an unseen task without environment samples, consequently finetuning with greater sample-efficiency. Furthermore, we generalize IRM to sequence skills for complex, long-horizon tasks and demonstrate that IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods on both the Fetch tabletop and Franka Kitchen robot manipulation benchmarks.

📄 PDF Abstract BibTeX arXiv:2210.07426

Code (1)

ademiadeniji/irm 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Robot ManipulationUnsupervised Reinforcement Learning

APART: Diverse Skill Discovery using All Pairs with Ascending Reward and DropouT

2023-08-24 · Hadar Schreiber Galler, Tom Zahavy, Guillaume Desjardins, Alon Cohen

We study diverse skill discovery in reward-free environments, aiming to discover all possible skills in simple grid-world environments where prior methods have struggled to succeed. This problem is formulated as mutual t…

All

CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

2022-02-01 · Michael Laskin, Hao liu, Xue Bin Peng, Denis Yarats 외

We introduce Contrastive Intrinsic Control (CIC), an algorithm for unsupervised skill discovery that maximizes the mutual information between state-transitions and latent skill vectors. CIC utilizes contrastive learning …

Contrastive Learning Diversity reinforcement-learning Reinforcement Learning (RL)+1

Constrained Intrinsic Motivation for Reinforcement Learning

2024-07-12 · Xiang Zheng, Xingjun Ma, Chao Shen, Cong Wang

This paper investigates two fundamental problems that arise when utilizing Intrinsic Motivation (IM) for reinforcement learning in Reward-Free Pre-Training (RFPT) tasks and Exploration with Intrinsic Motivation (EIM) tas…

MuJoCo reinforcement-learning Reinforcement Learning

Deep Reinforcement Learning with Hybrid Intrinsic Reward Model

2025-01-22 · Mingqi Yuan, Bo Li, Xin Jin, Wenjun Zeng

Intrinsic reward shaping has emerged as a prevalent approach to solving hard-exploration and sparse-rewards environments in reinforcement learning (RL). While single intrinsic rewards, such as curiosity-driven or novelty…

Deep Reinforcement Learning Diversity model reinforcement-learning+2

The impact of intrinsic rewards on exploration in Reinforcement Learning

2025-01-20 · Aya Kayal, Eduardo Pignatelli, Laura Toni

One of the open challenges in Reinforcement Learning is the hard exploration problem in sparse reward environments. Various types of intrinsic rewards have been proposed to address this challenge by pushing towards diver…

Diversity reinforcement-learning Reinforcement Learning Representation Learning