Learning Correlation Structures for Vision Transformers

2024-04-05 · CVPR 2024 1 · Manjin Kim, Paul Hongsuck Seo, Cordelia Schmid, Minsu Cho

We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate local contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effectiveness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-1K, Kinetics-400, Something-Something V1 & V2, Diving-48, and FineGym.

📄 PDF Abstract BibTeX arXiv:2404.03924

Code (0)

등록된 구현이 없습니다.

Tasks

Action Classification Action RecognitionObjectVideo Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Multi-Head Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

Are Vision Transformers Robust to Spurious Correlations?

2022-03-17 · Soumya Suvra Ghosal, Yifei Ming, Yixuan Li

Deep neural networks may be susceptible to learning spurious correlations that hold on average but not in atypical test samples. As with the recent emergence of vision transformer (ViT) models, it remains underexplored h…

A Recent Survey of Vision Transformers for Medical Image Segmentation

2023-12-01 · Asifullah Khan, Zunaira Rauf, Abdul Rehman Khan, Saima Rathore 외

Medical image segmentation plays a crucial role in various healthcare applications, enabling accurate diagnosis, treatment planning, and disease monitoring. Traditionally, convolutional neural networks (CNNs) dominated t…

Image Segmentation Inductive Bias Medical Image Segmentation Segmentation+2

Elastic ViTs from Pretrained Models without Retraining

2025-10-20 · Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G. M. Snoek 외 arxiv

Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: Single-s…

Mixed Transformer U-Net For Medical Image Segmentation

2021-11-08 · Hongyi Wang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto 외

Though U-Net has achieved tremendous success in medical image segmentation tasks, it lacks the ability to explicitly model long-range dependencies. Therefore, Vision Transformers have emerged as alternative segmentation …

Image Segmentation Medical Image Segmentation Segmentation Semantic Segmentation

Decorrelation Speeds Up Vision Transformers

2025-10-16 · Kieran Carrigg, Rob van Gastel, Melda Yeghaian, Sander Dalm 외 arxiv

Masked Autoencoder (MAE) pre-training of vision transformers (ViTs) yields strong performance in low-label data regimes but comes with substantial computational costs, making it impractical in time- and resource-constrai…