Hydra Attention: Efficient Attention with Many Heads

2022-09-15 · Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Judy Hoffman

While transformers have begun to dominate many tasks in vision, applying them to large images is still computationally difficult. A large reason for this is that self-attention scales quadratically with the number of tokens, which in turn, scales quadratically with the image size. On larger images (e.g., 1080p), over 60% of the total computation in the network is spent solely on creating and applying attention matrices. We take a step toward solving this issue by introducing Hydra Attention, an extremely efficient attention operation for Vision Transformers (ViTs). Paradoxically, this efficiency comes from taking multi-head attention to its extreme: by using as many attention heads as there are features, Hydra Attention is computationally linear in both tokens and features with no hidden constants, making it significantly faster than standard self-attention in an off-the-shelf ViT-B/16 by a factor of the token count. Moreover, Hydra Attention retains high accuracy on ImageNet and, in some cases, actually improves it.

📄 PDF Abstract BibTeX arXiv:2209.07484

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Hydra 설명 없음

HydraViT: Stacking Heads for a Scalable ViT

2024-09-26 · Janek Haberer, Ali Hojjat, Olaf Landsiedel

The architecture of Vision Transformers (ViTs), particularly the Multi-head Attention (MHA) mechanism, imposes substantial hardware demands. Deploying ViTs on devices with varying constraints, such as mobile phones, requ…

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

2026-06-18 · Zhentao Tan, Wei Chen, Jingyi Shen, Yao Liu 외 arxiv

The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work ha…

HYDRA -- Hyper Dependency Representation Attentions

2021-09-11 · Ha-Thanh Nguyen, Vu Tran, Tran-Binh Dang, Minh-Quan Bui 외

Attention is all we need as long as we have enough data. Even so, it is sometimes not easy to determine how much data is enough while the models are becoming larger and larger. In this paper, we propose HYDRA heads, ligh…

Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers

2025-10-21 · Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy 외 arxiv

Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs h…

Text Classification

Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding

2024-02-07 · Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard 외

To combat the memory bandwidth-bound nature of autoregressive LLM inference, previous research has proposed the speculative decoding frame-work. To perform speculative decoding, a small draft model proposes candidate con…