MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning
Recently, a state-of-the-art family of algorithms, known as Goal-Conditioned Weighted Supervised Learning (GCWSL) methods, has been introduced to tackle challenges in offline goal-conditioned reinforcement learning (RL). GCWSL optimizes a lower bound of the goal-conditioned RL objective and has demonstrated outstanding performance across diverse goal-reaching tasks, providing a simple, effective, and stable solution. However, prior research has identified a critical limitation of GCWSL: the lack of trajectory stitching capabilities. To address this, goal data augmentation strategies have been proposed to enhance these methods. Nevertheless, existing techniques often struggle to sample suitable augmented goals for GCWSL effectively. In this paper, we establish unified principles for goal data augmentation, focusing on goal diversity, action optimality, and goal reachability. Based on these principles, we propose a Model-based Goal Data Augmentation (MGDA) approach, which leverages a learned dynamics model to sample more suitable augmented goals. MGDA uniquely incorporates the local Lipschitz continuity assumption within the learned model to mitigate the impact of compounding errors. Empirical results show that MGDA significantly enhances the performance of GCWSL methods on both state-based and vision-based maze datasets, surpassing previous goal data augmentation techniques in improving stitching capabilities.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Goal-Conditioned Data Augmentation for Offline Reinforcement Learning
Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it genera…
D4RLData AugmentationOffline RLreinforcement-learning+3How to Solve Contextual Goal-Oriented Problems with Offline Datasets?
We present a novel method, Contextual goal-Oriented Data Augmentation (CODA), which uses commonly available unlabeled trajectories and context-goal pairs to solve Contextual Goal-Oriented (CGO) problems. By carefully con…
Data AugmentationTemporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
The goal of offline reinforcement learning (RL) is to extract a high-performance policy from the fixed datasets, minimizing performance degradation due to out-of-distribution (OOD) samples. Offline model-based RL (MBRL) …
D4RLModel-based Reinforcement LearningReinforcement Learning (RL)Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimization
In machine learning, the goal of multi-task learning (MTL) is to optimize multiple objectives together. Recent works, for example, Multiple Gradient Descent Algorithm (MGDA) and its variants, show promising results with …
Multi-Task LearningConflict-Averse Gradient Optimization of Ensembles for Effective Offline Model-Based Optimization
Data-driven offline model-based optimization (MBO) is an established practical approach to black-box computational design problems for which the true objective function is unknown and expensive to query. However, the sta…