When Do Skills Help Reinforcement Learning? A Theoretical Analysis of Temporal Abstractions

2024-06-12 · Zhening Li, Gabriel Poesia, Armando Solar-Lezama

Skills are temporal abstractions that are intended to improve reinforcement learning (RL) performance through hierarchical RL. Despite our intuition about the properties of an environment that make skills useful, a precise characterization has been absent. We provide the first such characterization, focusing on the utility of deterministic skills in deterministic sparse-reward environments with finite action spaces. We show theoretically and empirically that RL performance gain from skills is worse in environments where solutions to states are less compressible. Additional theoretical results suggest that skills benefit exploration more than they benefit learning from existing experience, and that using unexpressive skills such as macroactions may worsen RL performance. We hope our findings can guide research on automatic skill discovery and help RL practitioners better decide when and how to use skills.

📄 PDF Abstract BibTeX arXiv:2406.07897

Code (1)

uranium11010/rl-skill-theory 공식 구현 pytorch

Tasks

Reinforcement Learning (RL)

Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning

2025-06-12 · Yucheng Yang, Tianyi Zhou, Qiang He, Lei Han 외

Unsupervised reinforcement learning (URL) aims to learn general skills for unseen downstream tasks. Mutual Information Skill Learning (MISL) addresses URL by maximizing the mutual information between states and skills bu…

Disentanglement Diversity Unsupervised Reinforcement Learning

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

2017-12-20 · ICLR 2018 1 · Tianmin Shu, Caiming Xiong, Richard Socher

Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a n…

Minecraft reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards

2020-10-15 · Siyu Dai, Wei Xu, Andreas Hofmann, Brian Williams

In order to provide adaptive and user-friendly solutions to robotic manipulation, it is important that the agent can learn to accomplish tasks even if they are only provided with very sparse instruction signals. To addre…

Diversity reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Automata Guided Skill Composition

2019-05-01 · ICLR 2019 5 · Xiao Li, Yao Ma, Calin Belta

Skills learned through (deep) reinforcement learning often generalizes poorly across tasks and re-training is necessary when presented with a new task. We present a framework that combines techniques in formal methods wi…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

SkillFactory: Self-Distillation For Learning Cognitive Behaviors

2025-12-03 · Zayne Sprague, Jack Lu, Manya Wadhwa, Sedrick Keh 외 arxiv

Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base…

Reinforcement Learning