Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images

2024-02-06 · Jen Hong Tan

Can a lightweight Vision Transformer (ViT) match or exceed the performance of Convolutional Neural Networks (CNNs) like ResNet on small datasets with small image resolutions? This report demonstrates that a pure ViT can indeed achieve superior performance through pre-training, using a masked auto-encoder technique with minimal image scaling. Our experiments on the CIFAR-10 and CIFAR-100 datasets involved ViT models with fewer than 3.65 million parameters and a multiply-accumulate (MAC) count below 0.27G, qualifying them as 'lightweight' models. Unlike previous approaches, our method attains state-of-the-art performance among similar lightweight transformer-based architectures without significantly scaling up images from CIFAR-10 and CIFAR-100. This achievement underscores the efficiency of our model, not only in handling small datasets but also in effectively processing images close to their original scale.

📄 PDF Abstract BibTeX arXiv:2402.03752

Code (0)

등록된 구현이 없습니다.

Tasks

Image Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Residual Connection 설명 없음

Average Pooling 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

How Lightweight Can A Vision Transformer Be

2024-07-25 · Jen Hong Tan

In this paper, we explore a strategy that uses Mixture-of-Experts (MoE) to streamline, rather than augment, vision transformers. Each expert in an MoE layer is a SwiGLU feedforward network, where V and W2 are shared acro…

Mixture-of-Experts Transfer Learning

Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets

2022-10-25 · Xiangyu Chen, Ying Qin, Wenju Xu, Andrés M. Bur 외

Vision Transformers have attracted a lot of attention recently since the successful implementation of Vision Transformer (ViT) on vision tasks. With vision Transformers, specifically the multi-head self-attention modules…

AViT: Adapting Vision Transformers for Small Skin Lesion Segmentation Datasets

2023-07-26 · Siyi Du, Nourhan Bayasi, Ghassan Hamarneh, Rafeef Garbi

Skin lesion segmentation (SLS) plays an important role in skin lesion analysis. Vision transformers (ViTs) are considered an auspicious solution for SLS, but they require more training data compared to convolutional neur…

Lesion Segmentation Skin Lesion Segmentation

Multi-Scale Visual Prompting for Lightweight Small-Image Classification

2025-12-03 · Salim Khazem arxiv

Visual prompting has recently emerged as an efficient strategy to adapt vision models using lightweight, learnable parameters injected into the input space. However, prior work mainly targets large Vision Transformers an…

Image Classification

Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers

2026-06-08 · Leyla Naz Candogan, Arshia Afzal, Pol Puigdemont, Volkan Cevher arxiv

Though Vision Transformers (ViTs) have become the dominant backbone in many computer vision tasks, due to permutation equivariance, their attention mechanism lacks explicit spatial inductive biases. This become particula…

parameter-efficient fine-tuning