Rethinking Vision Transformer for Object Centric Foundation Models

2025-02-04 · Manuel Traub, Martin V. Butz

Recent state-of-the-art object segmentation mechanisms, such as the Segment Anything Model (SAM) and FastSAM, first encode the full image over several layers and then focus on generating the mask for one particular object or area. We present an off-grid Fovea-Like Input Patching (FLIP) approach, which selects image input and encodes it from the beginning in an object-focused manner. While doing so, it separates locational encoding from an object-centric perceptual code. FLIP is more data-efficient and yields improved segmentation performance when masking relatively small objects in high-resolution visual scenes. On standard benchmarks such as Hypersim, KITTI-360, and OpenImages, FLIP achieves Intersection over Union (IoU) scores that approach the performance of SAM with much less compute effort. It surpasses FastSAM in all IoU measurements. We also introduce an additional semi-natural but highly intuitive dataset where FLIP outperforms SAM and FastSAM overall and particularly on relatively small objects. Seeing that FLIP is an end-to-end object-centric segmentation approach, it has high potential particularly for applications that benefit from computationally efficient, spatially highly selective object tracking.

📄 PDF Abstract BibTeX arXiv:2502.02763

Code (1)

CognitiveModeling/FLIP pytorch

Tasks

ObjectObject Tracking Segmentation Semantic Segmentation

Methods 이 논문이 사용한 방법론

FLIP https://developer.nvidia.com/blog/flip-a-difference-evaluator-for-alternating-images/

Patching Activation patching studies the model's computation by altering its latent representations, the token embeddings in transformer-based language models, during the inference process

Focus 설명 없음

SAM 설명 없음

SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video

2024-06-13 · Hector A. Valdez, Kyle Min, Subarna Tripathi

Pretraining egocentric vision-language models has become essential to improving downstream egocentric video-text tasks. These egocentric foundation models commonly use the transformer architecture. The memory footprint o…

Data Augmentation

Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representation

2023-09-23 · ICCV 2023 1 · Ke Fan, Jingshi Lei, Xuelin Qian, Miaopeng Yu 외

Video amodal segmentation is a particularly challenging task in computer vision, which requires to deduce the full shape of an object from the visible parts of it. Recently, some studies have achieved promising performan…

Object Video Segmentation Video Semantic Segmentation

Learning Explicit Object-Centric Representations with Vision Transformers

2022-10-25 · Oscar Vikström, Alexander Ilin

With the recent successful adaptation of transformers to the vision domain, particularly when trained in a self-supervised fashion, it has been shown that vision transformers can learn impressive object-reasoning-like be…

Object Segmentation Semantic Segmentation

Human-like Object Grouping in Self-supervised Vision Transformers

2026-03-14 · Hossein Adeli, Seoyoung Ahn, Andrew Luo, Mengmi Zhang 외 arxiv

Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception …

Object Segmentation

VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

2025-10-23 · Jesimon Barreto, Carlos Caetano, André Araujo, William Robson Schwartz arxiv

Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution …

Self-Supervised Learning