Making Vision Transformers Truly Shift-Equivariant

2023-05-25 · CVPR 2024 1 · Renan A. Rojas-Gomez, Teck-Yian Lim, Minh N. Do, Raymond A. Yeh

For computer vision, Vision Transformers (ViTs) have become one of the go-to deep net architectures. Despite being inspired by Convolutional Neural Networks (CNNs), ViTs' output remains sensitive to small spatial shifts in the input, i.e., not shift invariant. To address this shortcoming, we introduce novel data-adaptive designs for each of the modules in ViTs, such as tokenization, self-attention, patch merging, and positional encoding. With our proposed modules, we achieve true shift-equivariance on four well-established ViTs, namely, Swin, SwinV2, CvT, and MViTv2. Empirically, we evaluate the proposed adaptive models on image classification and semantic segmentation tasks. These models achieve competitive performance across three different datasets while maintaining 100% shift consistency.

📄 PDF Abstract BibTeX arXiv:2305.16316

Code (0)

등록된 구현이 없습니다.

Tasks

image-classification Image Classification Semantic Segmentation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Residual Connection 설명 없음

Batch Normalization 설명 없음

Depthwise Separable Convolution While standard convolution performs the channelwise and spatial-wise computation in one step, Depthwise Separable Convolution …

Reviving Shift Equivariance in Vision Transformers

2023-06-13 · Peijian Ding, Davit Soselia, Thomas Armstrong, Jiahao Su 외

Shift equivariance is a fundamental principle that governs how we perceive the world - our recognition of an object remains invariant with respect to shifts. Transformers have gained immense popularity due to their effec…

Inductive Bias

REViT: Roto-reflection Equivariant Convolutional Vision Transformer

2026-06-24 · Sheir A. Zaheer, Alexander C. Holston, Chan Y. Park arxiv

In this paper, we propose a discrete roto-reflection group equivariant vision transformer with convolutional attention. Roto-reflection equivariant networks preserve the rotational, flip and positional symmetry in featur…

Image Classification Object Detection

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$

2026-06-26 · Tīkun Ông, Georg Bökman arxiv

Vision transformers have become a dominant architecture for visual recognition. However, standard models do not explicitly encode the planar symmetries that arise in many vision domains. We introduce a family of vision t…

Truly Scale-Equivariant Deep Nets with Fourier Layers

2023-11-06 · NeurIPS 2023 11 · Md Ashiqur Rahman, Raymond A. Yeh

In computer vision, models must be able to adapt to changes in image resolution to effectively carry out tasks such as image segmentation; This is known as scale-equivariance. Recent works have made progress in developin…

Image Segmentation Semantic Segmentation

Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks

2022-10-14 · Renan A. Rojas-Gomez, Teck-Yian Lim, Alexander G. Schwing, Minh N. Do 외

We propose learnable polyphase sampling (LPS), a pair of learnable down/upsampling layers that enable truly shift-invariant and equivariant convolutional networks. LPS can be trained end-to-end from data and generalizes …

image-classification Image Classification Segmentation Semantic Segmentation