Language-Mediated, Object-Centric Representation Learning
We present Language-mediated, Object-centric Representation Learning (LORL), a paradigm for learning disentangled, object-centric scene representations from vision and language. LORL builds upon recent advances in unsupervised object discovery and segmentation, notably MONet and Slot Attention. While these algorithms learn an object-centric representation just by reconstructing the input image, LORL enables them to further learn to associate the learned representations to concepts, i.e., words for object categories, properties, and spatial relationships, from language input. These object-centric concepts derived from language facilitate the learning of object-centric representations. LORL can be integrated with various unsupervised object discovery algorithms that are language-agnostic. Experiments show that the integration of LORL consistently improves the performance of unsupervised object discovery methods on two datasets via the help of language. We also show that concepts learned by LORL, in conjunction with object discovery methods, aid downstream tasks such as referring expression comprehension.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectObject DiscoveryReferring ExpressionReferring Expression ComprehensionRepresentation LearningSemantic SegmentationUnsupervised Object SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Room-Mediated Co-occurrence for Zero-Shot Object-Centric Semantic Navigation via Frontier Scoring
Zero-shot ObjectNav methods increasingly use vision-language priors, but direct object-object similarity in the latent space is often a weak proxy for spatial co-occurrence. We present an analytical, training-free semant…
System-Mediated Attention Imbalances Make Vision-Language Models Say Yes
Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric in…
CTRL-O: Language-Controllable Object-Centric Visual Representation Learning
Object-centric representation learning aims to decompose visual scenes into fixed-size vectors called "slots" or "object files", where each slot captures a distinct object. Current state-of-the-art object-centric models …
Image GenerationObjectObject DiscoveryQuestion Answering+4UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
Egocentric video understanding is inherently limited by the narrow perspective of wearable cameras: a single viewpoint, a single modality, a single model cannot capture the full richness of human action. We argue that a …
Representation LearningAction SegmentationAction RecognitionVideo RetrievalObject-Centric World Model for Language-Guided Manipulation
A world model is essential for an agent to predict the future and plan in domains such as autonomous driving and robotics. To achieve this, recent advancements have focused on video generation, which has gained significa…
Autonomous DrivingmodelObjectObject Recognition+1