SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers
Unsupervised object-centric learning aims to decompose scenes into interpretable object entities, termed slots. Slot-based auto-encoders stand out as a prominent method for this task. Within them, crucial aspects include guiding the encoder to generate object-specific slots and ensuring the decoder utilizes them during reconstruction. This work introduces two novel techniques, (i) an attention-based self-training approach, which distills superior slot-based attention masks from the decoder to the encoder, enhancing object segmentation, and (ii) an innovative patch-order permutation strategy for autoregressive transformers that strengthens the role of slot vectors in reconstruction. The effectiveness of these strategies is showcased experimentally. The combined approach significantly surpasses prior slot-based autoencoder methods in unsupervised object segmentation, especially with complex real-world images. We provide the implementation code at https://github.com/gkakogeorgiou/spot .
Code (1)
Tasks
DecoderObjectSegmentationSemantic SegmentationUnsupervised Object SegmentationSimilar Papers 제목 키워드 기반
Shuffle to Learn: Self-supervised learning from permutations via differentiable ranking
Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of tasks. In this work we advance self-supervised learning from permutations, that consists in sh…
General ClassificationSelf-Supervised LearningVideo ClassificationSelf-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking
Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of modalities. In this work, we advance self-supervised learning from permutations, by pre-traini…
ClassificationGeneral ClassificationSelf-Supervised LearningREOrdering Patches Improves Vision Models
Sequence models such as transformers require inputs to be represented as one-dimensional sequences. In vision, this typically involves flattening images using a fixed row-major (raster-scan) order. While full self-attent…
MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored. We introduce MUGEN, a comprehensive benchmark evaluating this capability across speech, general audio, and mu…
Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning
The development of autoregressive modeling (AM) in computer vision lags behind natural language processing (NLP) in self-supervised pre-training. This is mainly caused by the challenge that images are not sequential sign…
image-classificationImage ClassificationInstance Segmentationobject-detection+3