Offline Reinforcement Learning with Discrete Diffusion Skills

2025-03-26 · Ruixi Qiao, Jie Cheng, Xingyuan Dai, Yonglin Tian, Yisheng Lv

Skills have been introduced to offline reinforcement learning (RL) as temporal abstractions to tackle complex, long-horizon tasks, promoting consistent behavior and enabling meaningful exploration. While skills in offline RL are predominantly modeled within a continuous latent space, the potential of discrete skill spaces remains largely underexplored. In this paper, we propose a compact discrete skill space for offline RL tasks supported by state-of-the-art transformer-based encoder and diffusion-based decoder. Coupled with a high-level policy trained via offline RL techniques, our method establishes a hierarchical RL framework where the trained diffusion decoder plays a pivotal role. Empirical evaluations show that the proposed algorithm, Discrete Diffusion Skill (DDS), is a powerful offline RL method. DDS performs competitively on Locomotion and Kitchen tasks and excels on long-horizon tasks, achieving at least a 12 percent improvement on AntMaze-v2 benchmarks compared to existing offline RL approaches. Furthermore, DDS offers improved interpretability, training stability, and online exploration compared to previous skill-based methods.

📄 PDF Abstract BibTeX arXiv:2503.20176

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills

2023-09-24 · Zenan Li, Fan Nie, Qiao Sun, Fang Da 외

Learning-based vehicle planning is receiving increasing attention with the emergence of diverse driving simulators and large-scale driving datasets. While offline reinforcement learning (RL) is well suited for these safe…

Autonomous Driving Offline RL reinforcement-learning Reinforcement Learning (RL)

Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning

2025-07-09 · Jinmin He, Kai Li, Yifan Zang, Haobo Fu 외 arxiv

Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. Howe…

Reinforcement Learning

DiWA: Diffusion Policy Adaptation with World Models

2025-08-05 · Akshay L Chandra, Iman Nematollahi, Chenguang Huang, Tim Welschehold 외 arxiv

Fine-tuning diffusion policies with reinforcement learning (RL) presents significant challenges. The long denoising sequence for each action prediction impedes effective reward propagation. Moreover, standard RL methods …

Reinforcement Learning

Robust Policy Learning via Offline Skill Diffusion

2024-03-01 · Woo Kyung Kim, Minjong Yoo, Honguk Woo

Skill-based reinforcement learning (RL) approaches have shown considerable promise, especially in solving long-horizon tasks via hierarchical structures. These skills, learned task-agnostically from offline datasets, can…

Decoder Imitation Learning Reinforcement Learning (RL)

Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL

2024-02-11 · Sungyoon Kim, Yunseon Choi, Daiki E. Matsunaga, Kee-Eung Kim

Offline Goal-Conditioned Reinforcement Learning (Offline GCRL) is an important problem in RL that focuses on acquiring diverse goal-oriented skills solely from pre-collected behavior datasets. In this setting, the reward…

Offline RL