Self-supervised Object-Centric Learning for Videos
Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An additional modality such as depth or motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed in synthetic sequences, which rely on the robustness of an additional cue, do not translate to more challenging real-world scenarios. In this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in a high-level semantic feature space. We propose a masking strategy by dropping a significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address over-clustering by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectSemantic SegmentationSimilar Papers 제목 키워드 기반
Self-Supervised Object Detection from Egocentric Videos
Understanding the visual world from the perspective of humans (egocentric) has been a long-standing challenge in computer vision. Egocentric videos exhibit high scene complexity and irregular motion flows compared to…
Class-agnostic Object DetectionObjectobject-detectionObject Detection+2Slot-BERT: Self-supervised Object Discovery in Surgical Video
Object-centric slot attention is a powerful framework for unsupervised learning of structured and explainable representations that can support reasoning about objects and actions, including in surgical videos. While conv…
DisentanglementDomain AdaptationObjectObject DiscoveryVESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models
Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution …
Self-Supervised LearningUnsupervised Open-Vocabulary Object Localization in Videos
In this paper, we show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization. We propose a method tha…
ObjectObject LocalizationRepresentation Learning3D-DLP: Self-Supervised 3D Object-Centric Scene Representation Learning
We introduce 3D-DLP, a self-supervised object-centric representation learning model that decomposes scene-level RGB-D or voxel observations into a set of 3D latent particles. Building on the Deep Latent Particles (DLP) f…
Representation Learning