paper-with-me

홈 › Papers

Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

2026-03-09 · Shubham Aggarwal, Lokendra Kumar arxiv

The dense output projection in multi head attention scales quadratically with model dimension, contributing significantly to parameter count, memory footprint, and inference cost. We propose replacing this projection with a fixed, parameter free Walsh Hadamard Transform (WHT) followed by a diagonal affine transformation. This approach eliminates approximately 25 percent of attention parameters per block while maintaining global cross-head interaction through an orthogonal, norm-preserving transformation. Our results demonstrate that WHT augmented models exhibit a steeper validation loss curve relative to training FLOPs compared to dense baselines, suggesting superior compute utilization during training. Crucially, we show that efficiency gains including reduced memory footprint and increased throughput grow monotonically with model size, batch size, and sequence length. We evaluate performance across both prefill and decoding stages, finding that the structured transform consistently outperforms dense projections as complexity increases. Our findings indicate that replacing dense projections with structured transforms allows for more compute-efficient architectures that achieve lower loss than dense models at an equivalent training budget.

📄 PDF Abstract BibTeX arXiv:2603.08343

Code (0)

등록된 구현이 없습니다.

Similar Papers 제목 키워드 기반

Adversarial Attack and Disturbance Detection by Hadamard-Coded Output Representations for Object Detection and Semantic Segmentation

2026-06-08 · Lucas Görnhardt, Timo Bartels, Niklas Schwarz, Tim Fingscheidt arxiv

Conventional one-hot encodings often yield poorly calibrated models, being overconfident under attack, and letting entropy-based detection algorithms fail. Previous image classification works have demonstrated that Hadam…

Adversarial RobustnessSemantic SegmentationImage ClassificationAdversarial Attack

Visual Explanations from Hadamard Product in Multimodal Deep Networks

2017-12-18 · Jin-Hwa Kim, Byoung-Tak Zhang

The visual explanation of learned representation of models helps to understand the fundamentals of learning. The attentional models of previous works used to visualize the attended regions over an image or text using the…

Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)

Hadamard Codebook Based Deep Hashing

2019-10-21 · Shen Chen, Liujuan Cao, Mingbao Lin, Yan Wang 외

As an approximate nearest neighbor search technique, hashing has been widely applied in large-scale image retrieval due to its excellent efficiency. Most supervised deep hashing methods have similar loss designs with emb…

Deep HashingImage RetrievalRetrieval

Improvements of computational ghost imaging by using Special-Hadamard patterns

2019-02-12

We introduced a new kind of patterns named Special-Hadamard patterns, which could be used as structured illuminations of computational ghost imaging. Special-Hadamard patterns can get a better image quality than Hadamard…

Object

Optimal Iterative Sketching with the Subsampled Randomized Hadamard Transform

2020-02-03 · Jonathan Lacotte, Sifan Liu, Edgar Dobriban, Mert Pilanci

Random projections or sketching are widely used in many algorithmic and learning contexts. Here we study the performance of iterative Hessian sketch for least-squares problems. By leveraging and extending recent results …

Dimensionality Reduction