Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks
Goal-conditioned (GC) policy learning often faces a challenge arising from the sparsity of rewards, when confronting long-horizon goals. To address the challenge, we explore skill-based GC policy learning in offline settings, where skills are acquired from existing data and long-horizon goals are decomposed into sequences of near-term goals that align with these skills. Specifically, we present an `offline GC policy learning via skill-step abstraction' framework (GLvSA) tailored for tackling long-horizon GC tasks affected by goal distribution shifts. In the framework, a GC policy is progressively learned offline in conjunction with the incremental modeling of skill-step abstractions on the data. We also devise a GC policy hierarchy that not only accelerates GC policy learning within the framework but also allows for parameter-efficient fine-tuning of the policy. Through experiments with the maze and Franka kitchen environments, we demonstrate the superiority and efficiency of our GLvSA framework in adapting GC policies to a wide range of long-horizon goals. The framework achieves competitive zero-shot and few-shot adaptation performance, outperforming existing GC policy learning and skill-based methods.
Code (0)
등록된 구현이 없습니다.
Tasks
parameter-efficient fine-tuningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning
Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. Howe…
Reinforcement LearningOffline Reinforcement Learning with Discrete Diffusion Skills
Skills have been introduced to offline reinforcement learning (RL) as temporal abstractions to tackle complex, long-horizon tasks, promoting consistent behavior and enabling meaningful exploration. While skills in offlin…
DecoderOffline RLreinforcement-learningReinforcement Learning+1Divide and Conquer: Grounding LLMs as Efficient Decision-Making Agents via Offline Hierarchical Reinforcement Learning
While showing sophisticated reasoning abilities, large language models (LLMs) still struggle with long-horizon decision-making tasks due to deficient exploration and long-term credit assignment, especially in sparse-rewa…
Decision MakingHierarchical Reinforcement LearningBoosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills
Learning-based vehicle planning is receiving increasing attention with the emergence of diverse driving simulators and large-scale driving datasets. While offline reinforcement learning (RL) is well suited for these safe…
Autonomous DrivingOffline RLreinforcement-learningReinforcement Learning (RL)Symskill: Symbol and Skill Co-Invention for Data-Efficient and Reactive Long-Horizon Manipulation
Multi-step manipulation in dynamic environments remains challenging. Imitation learning (IL) is reactive but lacks compositional generalization, since monolithic policies do not decide which skill to reuse when scenes ch…
Motion Planning