Toward Compositional Generalization in Object-Oriented World Modeling
Compositional generalization is a critical ability in learning and decision-making. We focus on the setting of reinforcement learning in object-oriented environments to study compositional generalization in world modeling. We (1) formalize the compositional generalization problem with an algebraic approach and (2) study how a world model can achieve that. We introduce a conceptual environment, Object Library, and two instances, and deploy a principled pipeline to measure the generalization ability. Motivated by the formulation, we analyze several methods with exact or no compositional generalization ability using our framework, and design a differentiable approach, Homomorphic Object-oriented World Model (HOWM), that achieves soft but more efficient compositional generalization.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingObjectreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
OBJECT-ORIENTED REPRESENTATION OF 3D SCENES
In this paper, we propose a generative model, called ROOTS (Representation of Object-Oriented Three-dimension Scenes), for unsupervised object-wise 3D-scene decomposition and and rendering. For 3D scene modeling, ROOTS b…
DisentanglementObjectBuilding Object-based Causal Programs for Human-like Generalization
We present a novel task that measures how people generalize objects' causal powers based on observing a single (Experiment 1) or a few (Experiment 2) causal interactions between object pairs. We propose a computational m…
NavigateObjectWhat Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives
Compositional generalization, the ability to generate novel combinations of known concepts, is a key ingredient for visual generative models. Yet, not all mechanisms that enable or inhibit it are fully understood. In thi…
Video GenerationPretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
We investigate the success conditions for compositional generalization of CLIP models on real-world data through performance prediction. Prior work shows that CLIP requires exponentially more pretraining data for linear …
When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks
Humans can reason compositionally whilst grounding language utterances to the real world. Recent benchmarks like ReaSCAN use navigation tasks grounded in a grid world to assess whether neural models exhibit similar capab…
Compositional Generalization (AVG)