Rethinking Low-Rank Adaptation in Vision: Exploring Head-Level Responsiveness across Diverse Tasks

2024-04-13 · Yibo Zhong, Yao Zhou

Low-rank adaptation (LoRA) has shifted the paradigm of adapting pre-trained Vision Transformers (ViT), achieving great efficiency by updating only a subset of tailored parameters to approximate weight updates. However, the multi-head design of the self-attention mechanism, with the heads working in parallel in the computation flow, exhibiting similar visual patterns and requiring update over all of them, incurs unnecessary storage and computational overhead. In this paper, we propose Head-level responsiveness tuning for low-rank adaptation (Heart-LoRA). The proposed method explores redundancy among the heads and selectively activates task-responsive heads, thus enabling fine-grained head-level tuning. Additionally, given the different responsiveness of heads to diverse visual tasks, our proposed method dynamically activates a subset of the approximated heads that are tailored to the current task. Experimental results show that Heart-LoRA yields superior performance over state-of-the-art PETL approaches on visual adaptation benchmark datasets.

📄 PDF Abstract BibTeX arXiv:2404.08894

Code (0)

등록된 구현이 없습니다.

Tasks

Transfer Learning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Adam 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads

2025-10-05 · Nghiem T. Diep, Dung Le, Tuan Truong, Tan Dinh 외 arxiv

Parameter-efficient fine-tuning (PEFT) has emerged as a powerful paradigm for adapting large-scale pre-trained models to downstream tasks with minimal additional parameters. Among PEFT methods, Low-Rank Adaptation (LoRA)…

parameter-efficient fine-tuning

Selective LoRA for Visual Tokens and Attention Heads

2025-12-22 · Tiange Luo, Lajanugen Logeswaran, Jaekyeom Kim, Justin Johnson 외 arxiv

Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, but its standard all-token, all-head design ignores the heterogeneous structure of vision language model (VLM) inputs. We introduce \emph{Ima…

parameter-efficient fine-tuning Visual Localization

Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment

2026-02-02 · Hongyi Peng, Han Yu, Xiaoxiao Li, Qiang Yang arxiv

Low-Rank Adaptation (LoRA) is widely used for federated fine-tuning. Yet under non-IID settings, it can substantially underperform full-parameter fine-tuning. Through with-high-probability robustness analysis, we uncover…

Federated Learning

Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers

2025-12-31 · Zheng Liu, Jinchao Zhu, Gao Huang arxiv

Low-rank adaptation (LoRA) has achieved remarkable success in fine-tuning pre-trained vision transformers for various downstream tasks. Existing studies mainly focus on exploring more parameter-efficient strategies or mo…

Representation Learning

ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs

2025-06-23 · Haseeb Ullah Khan Shinwari, Muhammad Usama

Conventional Low-Rank Adaptation (LoRA) methods employ a fixed rank, imposing uniform adaptation across transformer layers and attention heads despite their heterogeneous learning dynamics. This paper introduces Adaptive…