Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

2019-07-18 · ICLR 2020 1 · Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, Sergey Levine

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://sites.google.com/view/dynamical-distance-learning.

📄 PDF Abstract BibTeX arXiv:1907.08225

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Controllability-Aware Unsupervised Skill Discovery

2023-02-10 · Seohong Park, Kimin Lee, Youngwoon Lee, Pieter Abbeel

One of the key capabilities of intelligent agents is the ability to discover useful skills without external supervision. However, the current unsupervised skill discovery methods are often limited to acquiring simple, ea…

TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations

2024-07-11 · Junik Bae, Kwanyoung Park, Youngwoon Lee

Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover …

SUSD: Structured Unsupervised Skill Discovery through State Factorization

2026-02-02 · Seyed Mohammad Hadi Hosseini, Mahdieh Soleymani Baghshah arxiv

Unsupervised Skill Discovery (USD) aims to autonomously learn a diverse set of skills without relying on extrinsic rewards. One of the most common USD approaches is to maximize the Mutual Information (MI) between skill l…

Hierarchical Reinforcement Learning

Periodic Skill Discovery

2025-11-05 · Jonghae Park, Daesol Cho, Jusuk Lee, Dongseok Shim 외 arxiv

Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying on external rewards. However, current methods often overlook the periodic nature of learned skills, focusing ins…

Reinforcement Learning

Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning

2025-06-12 · Yucheng Yang, Tianyi Zhou, Qiang He, Lei Han 외

Unsupervised reinforcement learning (URL) aims to learn general skills for unseen downstream tasks. Mutual Information Skill Learning (MISL) addresses URL by maximizing the mutual information between states and skills bu…

Disentanglement Diversity Unsupervised Reinforcement Learning