APLA: A Simple Adaptation Method for Vision Transformers

2025-03-14 · Moein Sorkhei, Emir Konuk, Kevin Smith, Christos Matsoukas

Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at https://github.com/MoeinSorkhei/APLA.

📄 PDF Abstract BibTeX arXiv:2503.11335

Code (1)

moeinsorkhei/apla 공식 구현 pytorch

Tasks

ClassificationGPUScene Classification

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements

2025-07-28 · Aditya Ravuri, Neil D. Lawrence arxiv

We propose a probabilistic interpretation of transformers as unrolled inference steps assuming a probabilistic Laplacian Eigenmaps model from the ProbDR framework. Our derivation shows that at initialisation, transformer…

Dimensionality Reduction

Learning Explicit Object-Centric Representations with Vision Transformers

2022-10-25 · Oscar Vikström, Alexander Ilin

With the recent successful adaptation of transformers to the vision domain, particularly when trained in a self-supervised fashion, it has been shown that vision transformers can learn impressive object-reasoning-like be…

Object Segmentation Semantic Segmentation

Laplacian-LoRA: Delaying Oversmoothing in Deep GCNs via Spectral Low-Rank Adaptation

2026-02-07 · Sai Vamsi Alisetti arxiv

Oversmoothing is a fundamental limitation of deep graph convolutional networks (GCNs), causing node representations to collapse as depth increases. While many prior approaches mitigate this effect through architectural m…

Accelerated Linearized Laplace Approximation for Bayesian Deep Learning

2022-10-23 · Zhijie Deng, Feng Zhou, Jun Zhu

Laplace approximation (LA) and its linearized variant (LLA) enable effortless adaptation of pretrained deep neural networks to Bayesian neural networks. The generalized Gauss-Newton (GGN) approximation is typically intro…

Deep Learning

Laplacian Heads Improve Transformers by Smoothing Token Representations

2026-02-10 · Yuchong Zhang, Vardan Papyan arxiv

Transformers update token representations through multi-head attention and residual connections as $X \leftarrow X + \sum_{i} P^{(i)}XW_{V_i}W_{o_i}$, where $P^{(i)}$ is the softmax attention matrix in head $i$. We propo…

Self-Supervised Learning