Convolutional Initialization for Data-Efficient Vision Transformers

2024-01-23 · Jianqiao Zheng, Xueqian Li, Simon Lucey

Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging their architectural inductive bias. In this paper, we investigate whether this inductive bias can be reinterpreted as an initialization bias within a vision transformer network. Our approach is motivated by the finding that random impulse filters can achieve almost comparable performance to learned filters in CNNs. We introduce a novel initialization strategy for transformer networks that can achieve comparable performance to CNNs on small datasets while preserving its architectural flexibility.

📄 PDF Abstract BibTeX arXiv:2401.12511

Code (1)

osiriszjq/impulse_init 공식 구현 pytorch

Tasks

Inductive Bias

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Residual Connection 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition

2021-06-09 · Ziyuan Huang, Zhiwu Qing, Xiang Wang, Yutong Feng 외

With the recent surge in the research of vision transformers, they have demonstrated remarkable potential for various challenging computer vision applications, such as image recognition, point cloud classification as wel…

Action Recognition Point Cloud Classification Prediction Video Understanding

Structured Initialization for Attention in Vision Transformers

2024-04-01 · Jianqiao Zheng, Xueqian Li, Simon Lucey

The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform wel…

Inductive Bias

Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

2024-05-22 · Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Ahmet Enis Cetin 외

Central to the Transformer architectures' effectiveness is the self-attention mechanism, a function that maps queries, keys, and values into a high-dimensional vector space. However, training the attention weights of que…

Quantization

Effective Theory of Transformers at Initialization

2023-04-04 · Emily Dinan, Sho Yaida, Susan Zhang

We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This an…

Mimetic Initialization of Self-Attention Layers

2023-05-16 · Asher Trockman, J. Zico Kolter

It is notoriously difficult to train Transformers on small datasets; typically, large pre-trained models are instead used as the starting point. We explore the weights of such pre-trained Transformers (particularly for v…