What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?

2023-05-30 · Rui Yang, Yong Lin, Xiaoteng Ma, Hao Hu, Chongjie Zhang, Tong Zhang

Offline goal-conditioned RL (GCRL) offers a way to train general-purpose agents from fully offline datasets. In addition to being conservative within the dataset, the generalization ability to achieve unseen goals is another fundamental challenge for offline GCRL. However, to the best of our knowledge, this problem has not been well studied yet. In this paper, we study out-of-distribution (OOD) generalization of offline GCRL both theoretically and empirically to identify factors that are important. In a number of experiments, we observe that weighted imitation learning enjoys better generalization than pessimism-based offline RL method. Based on this insight, we derive a theory for OOD generalization, which characterizes several important design choices. We then propose a new offline GCRL method, Generalizable Offline goAl-condiTioned RL (GOAT), by combining the findings from our theoretical and empirical studies. On a new benchmark containing 9 independent identically distributed (IID) tasks and 17 OOD tasks, GOAT outperforms current state-of-the-art methods by a large margin.

📄 PDF Abstract BibTeX arXiv:2305.18882

Code (1)

yangrui2015/goat 공식 구현 tf

Tasks

Imitation LearningOffline RL

Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning

2026-05-20 · Junseok Kim, Dohyeong Kim, Mineui Hong, Songhwai Oh arxiv

Compositional generalization is essential for reaching unseen goals under novel contextual variations in offline goal-conditioned reinforcement learning (GCRL), where a generalist goal-reaching agent must be learned from…

Reinforcement Learning

Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Tasks

2022-10-12 · Kuan Fang, Patrick Yin, Ashvin Nair, Homer Walke 외

The utilization of broad datasets has proven to be crucial for generalization for a wide range of fields. However, how to effectively make use of diverse multi-task data for novel downstream tasks still remains a grand c…

GraSP-STL: A Graph-Based Framework for Zero-Shot Signal Temporal Logic Planning via Offline Goal-Conditioned Reinforcement Learning

2026-03-31 · Ancheng Hou, Ruijia Liu, Xiang Yin arxiv

This paper studies offline, zero-shot planning under Signal Temporal Logic (STL) specifications. We assume access only to an offline dataset of state-action-state transitions collected by a task-agnostic behavior policy,…

Zero-shot Generalization Reinforcement Learning

LLMs for Generalizable Language-Conditioned Policy Learning under Minimal Data Requirements

2024-12-09 · Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar

To develop autonomous agents capable of executing complex, multi-step decision-making tasks as specified by humans in natural language, existing reinforcement learning approaches typically require expensive labeled datas…

Decision Making Instruction Following

RvS: What is Essential for Offline RL via Supervised Learning?

2021-12-20 · Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, Sergey Levine

Recent work has shown that supervised learning alone, without temporal difference (TD) learning, can be remarkably effective for offline RL. When does this hold true, and which algorithmic components are necessary? Throu…

Offline RL