Single-Layer Vision Transformers for More Accurate Early Exits with Less Overhead

2021-05-19 · Arian Bakhtiarnia, Qi Zhang, Alexandros Iosifidis

Deploying deep learning models in time-critical applications with limited computational resources, for instance in edge computing systems and IoT networks, is a challenging task that often relies on dynamic inference methods such as early exiting. In this paper, we introduce a novel architecture for early exiting based on the vision transformer architecture, as well as a fine-tuning strategy that significantly increase the accuracy of early exit branches compared to conventional approaches while introducing less overhead. Through extensive experiments on image and audio classification as well as audiovisual crowd counting, we show that our method works for both classification and regression problems, and in both single- and multi-modal settings. Additionally, we introduce a novel method for integrating audio and visual modalities within early exits in audiovisual data analysis, that can lead to a more fine-grained dynamic inference.

📄 PDF Abstract BibTeX arXiv:2105.09121

Code (0)

등록된 구현이 없습니다.

Tasks

Audio Classification Crowd CountingEdge-computing

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformers

2025-11-13 · Minjun Kim, Jaeri Lee, Jongjin Kim, Jeongin Yun 외 arxiv

How can we accurately quantize a pre-trained Vision Transformer model? Quantization algorithms compress Vision Transformers (ViTs) into low-bit formats, reducing memory and computation demands with minimal accuracy degra…

Image Classification Object Detection

Can Vision Transformers Perform Convolution?

2021-11-02 · Shanda Li, Xiangning Chen, Di He, Cho-Jui Hsieh

Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional l…

SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity

2024-09-13 · Qitian Wu, Kai Yang, Hengrui Zhang, David Wipf 외

Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturin…

Deep Attention Representation Learning

GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformers

2022-01-16 · ACL ARR January 2022 1 · Anonymous

There has been a growing interest in interpreting the underlying dynamics of Transformers. While self-attention patterns were initially deemed as the primary choice, recent studies have shown that integrating other compo…

Advancing Video Anomaly Detection: A Bi-Directional Hybrid Framework for Enhanced Single- and Multi-Task Approaches

2025-04-20 · Guodong Shen, Yuqi Ouyang, Junru Lu, Yixuan Yang 외

Despite the prevailing transition from single-task to multi-task approaches in video anomaly detection, we observe that many adopt sub-optimal frameworks for individual proxy tasks. Motivated by this, we contend that opt…

Anomaly Detection Video Anomaly Detection