Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning

2023-06-14 · Ce Hao, Catherine Weaver, Chen Tang, Kenta Kawamoto, Masayoshi Tomizuka, Wei Zhan

Hierarchical reinforcement learning (RL) can accelerate long-horizon decision-making by temporally abstracting a policy into multiple levels. Promising results in sparse reward environments have been seen with skills, i.e. sequences of primitive actions. Typically, a skill latent space and policy are discovered from offline data. However, the resulting low-level policy can be unreliable due to low-coverage demonstrations or distribution shifts. As a solution, we propose the Skill-Critic algorithm to fine-tune the low-level policy in conjunction with high-level skill selection. Our Skill-Critic algorithm optimizes both the low-level and high-level policies; these policies are initialized and regularized by the latent space learned from offline demonstrations to guide the parallel policy optimization. We validate Skill-Critic in multiple sparse-reward RL environments, including a new sparse-reward autonomous racing task in Gran Turismo Sport. The experiments show that Skill-Critic's low-level policy fine-tuning and demonstration-guided regularization are essential for good performance. Code and videos are available at our website: https://sites.google.com/view/skill-critic.

📄 PDF Abstract BibTeX arXiv:2306.08388

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous RacingDecision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models

2021-11-25 · Iman Nematollahi, Erick Rosete-Beas, Adrian Röfer, Tim Welschehold 외

A core challenge for an autonomous agent acting in the real world is to adapt its repertoire of skills to cope with its noisy perception and dynamics. To scale learning of skills to long-horizon tasks, robots should be a…

Deep Reinforcement Learning

Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

2017-12-20 · ICLR 2018 1 · Tianmin Shu, Caiming Xiong, Richard Socher

Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a n…

Minecraft reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolution

2026-09-04 · Jinyuan Feng, Dongmin Li, Yiqun Chen, Yang Gao 외 arxiv

Skill libraries improve the sample efficiency of agentic reinforcement learning (RL) by enabling large language model (LLM) agents to reuse procedural knowledge. Yet existing paradigms exhibit structural shortcomings: th…

Reinforcement Learning

Situational Awareness by Risk-Conscious Skills

2016-10-10 · Daniel J. Mankowitz, Aviv Tamar, Shie Mannor

Hierarchical Reinforcement Learning has been previously shown to speed up the convergence rate of RL planning algorithms as well as mitigate feature-based model misspecification (Mankowitz et. al. 2016a,b, Bacon 2015). T…

Hierarchical Reinforcement Learning Reinforcement Learning

Weakly-Supervised Learning of Disentangled and Interpretable Skills for Hierarchical Reinforcement Learning

2021-09-29 · Wonil Song, Sangryul Jeon, Hyesong Choi, Kwanghoon Sohn 외

Hierarchical reinforcement learning (RL) usually requires task-agnostic and interpretable skills that can be applicable to various downstream tasks. While many recent works have been proposed to learn such skills for a p…

Decoder Hierarchical Reinforcement Learning Inductive Bias reinforcement-learning+2