Goal-Conditioned Data Augmentation for Offline Reinforcement Learning
Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it generally fails to learn well-qualified policies in suboptimal datasets. To address datasets with insufficient optimal demonstrations, we introduce Goal-cOnditioned Data Augmentation (GODA), a novel goal-conditioned diffusion-based method for augmenting samples with higher quality. Leveraging recent advancements in generative modeling, GODA incorporates a novel return-oriented goal condition with various selection mechanisms. Specifically, we introduce a controllable scaling technique to provide enhanced return-based guidance during data sampling. GODA learns a comprehensive distribution representation of the original offline datasets while generating new data with selectively higher-return goals, thereby maximizing the utility of limited optimal demonstrations. Furthermore, we propose a novel adaptive gated conditioning method for processing noised inputs and conditions, enhancing the capture of goal-oriented guidance. We conduct experiments on the D4RL benchmark and real-world challenges, specifically traffic signal control (TSC) tasks, to demonstrate GODA's effectiveness in enhancing data quality and superior performance compared to state-of-the-art data augmentation methods across various offline RL algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
D4RLData AugmentationOffline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Traffic Signal ControlSimilar Papers 제목 키워드 기반
MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning
Recently, a state-of-the-art family of algorithms, known as Goal-Conditioned Weighted Supervised Learning (GCWSL) methods, has been introduced to tackle challenges in offline goal-conditioned reinforcement learning (RL).…
Data AugmentationReinforcement Learning (RL)Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
The goal of offline reinforcement learning (RL) is to extract a high-performance policy from the fixed datasets, minimizing performance degradation due to out-of-distribution (OOD) samples. Offline model-based RL (MBRL) …
D4RLModel-based Reinforcement LearningReinforcement Learning (RL)Swapped goal-conditioned offline reinforcement learning
Offline goal-conditioned reinforcement learning (GCRL) can be challenging due to overfitting to the given dataset. To generalize agents' skills outside the given dataset, we propose a goal-swapping procedure that generat…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
Offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data. Despite recent advances in hierarchical ar…
Reinforcement LearningRethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL
Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Condi…
Offline RLReinforcement Learning (RL)Self-Supervised Learning