$E(2)$-Equivariant Vision Transformer

2023-06-11 · Renjun Xu, Kaifan Yang, Ke Liu, Fengxiang He

Vision Transformer (ViT) has achieved remarkable performance in computer vision. However, positional encoding in ViT makes it substantially difficult to learn the intrinsic equivariance in data. Initial attempts have been made on designing equivariant ViT but are proved defective in some cases in this paper. To address this issue, we design a Group Equivariant Vision Transformer (GE-ViT) via a novel, effective positional encoding operator. We prove that GE-ViT meets all the theoretical requirements of an equivariant neural network. Comprehensive experiments are conducted on standard benchmark datasets, demonstrating that GE-ViT significantly outperforms non-equivariant self-attention networks. The code is available at https://github.com/ZJUCDSYangKaifan/GEVit.

📄 PDF Abstract BibTeX arXiv:2306.06722

Code (1)

zjucdsyangkaifan/gevit 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Residual Connection 설명 없음

REViT: Roto-reflection Equivariant Convolutional Vision Transformer

2026-06-24 · Sheir A. Zaheer, Alexander C. Holston, Chan Y. Park arxiv

In this paper, we propose a discrete roto-reflection group equivariant vision transformer with convolutional attention. Roto-reflection equivariant networks preserve the rotational, flip and positional symmetry in featur…

Image Classification Object Detection

A Unified Framework for Vision Transformers Equivariant to Discrete Subgroups of $\mathrm{O}(2)$

2026-06-26 · Tīkun Ông, Georg Bökman arxiv

Vision transformers have become a dominant architecture for visual recognition. However, standard models do not explicitly encode the planar symmetries that arise in many vision domains. We introduce a family of vision t…

Vanilla Group Equivariant Vision Transformer: Simple and Effective

2026-02-08 · Jiahong Fu, Qi Xie, Deyu Meng, Zongben Xu arxiv

Incorporating symmetry priors as inductive biases to design equivariant Vision Transformers (ViTs) has emerged as a promising avenue for enhancing their performance. However, existing equivariant ViTs often struggle to b…

Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance

2023-06-05 · NeurIPS 2023 11 · Jinwoo Kim, Tien Dat Nguyen, Ayhan Suleymanzade, Hyeokjun An 외

We present a novel framework to overcome the limitations of equivariant architectures in learning functions with group symmetries. In contrary to equivariant architectures, we use an arbitrary base model such as an MLP o…

Graph Classification Graph Regression Link Prediction

Equi-ViT: Rotational Equivariant Vision Transformer for Robust Histopathology Analysis

2026-01-14 · Fuyao Chen, Yuexi Du, Elèonore V. Lieffrig, Nicha C. Dvornek 외 arxiv

Vision Transformers (ViTs) have gained rapid adoption in computational pathology for their ability to model long-range dependencies through self-attention, addressing the limitations of convolutional neural networks that…