Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation

2025-08-09 · Xiao Huang, Xu Liu, Enze Zhang, Tong Yu, Shuai Li arxiv

Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing work used offline datasets to generate data that conform to the online data distribution for data augmentation. However, generated data still exhibits a gap with the online data, limiting overall performance. To address this, we propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier-free guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, it employs a reweighting method to enable more generated data to align with the online data, enhancing performance while maintaining the agent's stability. Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15% average improvement in empirical performance on the D4RL benchmark such as MuJoCo and AntMaze.

📄 PDF Abstract BibTeX arXiv:2508.06806

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Data Augmentation

GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning

2024-05-27 · Jaewoo Lee, Sujin Yun, Taeyoung Yun, Jinkyoo Park

Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection a…

Data Augmentation Decision Making Offline RL reinforcement-learning+1

Prior-Guided Diffusion Planning for Offline Reinforcement Learning

2025-05-16 · Donghyeon Ki, JunHyeok Oh, Seong-Woong Shim, Byung-Jun Lee

Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilita…

Decision Making Denoising Offline RL reinforcement-learning+1

Adding Conditional Control to Diffusion Models with Reinforcement Learning

2024-06-17 · Yulai Zhao, Masatoshi Uehara, Gabriele Scalia, Tommaso Biancalani 외

Diffusion models are powerful generative models that allow for precise control over the characteristics of the generated samples. While these diffusion models trained on large datasets have achieved success, there is oft…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Guiding Online Reinforcement Learning with Action-Free Offline Pretraining

2023-01-30 · Deyao Zhu, Yuhui Wang, Jürgen Schmidhuber, Mohamed Elhoseiny

Offline RL methods have been shown to reduce the need for environment interaction by training agents using offline collected episodes. However, these methods typically require action information to be logged during data …

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning

2026-05-29 · Ha Manh Bui, Metod Jazbec, Eric Nalisnick, Anqi Liu arxiv

Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distr…

Reinforcement Learning