Exploring and Improving Mobile Level Vision Transformers

2021-08-30 · Pengguang Chen, Yixin Chen, Shu Liu, MingChang Yang, Jiaya Jia

We study the vision transformer structure in the mobile level in this paper, and find a dramatic performance drop. We analyze the reason behind this phenomenon, and propose a novel irregular patch embedding module and adaptive patch fusion module to improve the performance. We conjecture that the vision transformer blocks (which consist of multi-head attention and feed-forward network) are more suitable to handle high-level information than low-level features. The irregular patch embedding module extracts patches that contain rich high-level information with different receptive fields. The transformer blocks can obtain the most useful information from these irregular patches. Then the processed patches pass the adaptive patch merging module to get the final features for the classifier. With our proposed improvements, the traditional uniform vision transformer structure can achieve state-of-the-art results in mobile level. We improve the DeiT baseline by more than 9\% under the mobile-level settings and surpass other transformer architectures like Swin and CoaT by a large margin.

📄 PDF Abstract BibTeX arXiv:2108.13015

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Residual Connection 설명 없음

Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices

2023-06-20 · Ioannis Panopoulos, Sokratis Nikolaidis, Stylianos I. Venieris, Iakovos S. Venieris

Deep learning (DL) is characterised by its dynamic nature, with new deep neural network (DNN) architectures and approaches emerging every few years, driving the field's advancement. At the same time, the ever-increasing …

Rethinking Vision Transformers for MobileNet Size and Speed

2022-12-15 · ICCV 2023 1 · Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis 외

With the success of Vision Transformers (ViTs) in computer vision tasks, recent arts try to optimize the performance and complexity of ViTs to enable efficient deployment on mobile devices. Multiple approaches are propos…

RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone

2024-12-14 · Mustafa Munir, Md Mostafijur Rahman, Radu Marculescu

Vision transformers (ViTs) have dominated computer vision in recent years. However, ViTs are computationally expensive and not well suited for mobile devices; this led to the prevalence of convolutional neural network (C…

image-classification Image Classification Instance Segmentation object-detection+2

Separable Self-attention for Mobile Vision Transformers

2022-06-06 · Sachin Mehta, Mohammad Rastegari

Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latenc…

Image Classification Object Detection

Computer Vision Accelerators for Mobile Systems based on OpenCL GPGPU Co-Processing

2014-03-17 · Guohui Wang, Yingen Xiong, Jay Yun, Joseph R. Cavallaro

In this paper, we present an OpenCL-based heterogeneous implementation of a computer vision algorithm -- image inpainting-based object removal algorithm -- on mobile devices. To take advantage of the computation power of…

CPU GPU Image Inpainting