Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activations
Activation functions critically influence trainability and expressivity, and recent work has therefore explored a broad range of nonlinearities. However, widely used Gaussian i.i.d. initializations are designed to preser…
TLoRA: Task-aware Low Rank Adaptation of Large Language Models
Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well a…
parameter-efficient fine-tuningNatural Language UnderstandingCode GenerationDetecting Adversarial Samples Using Influence Functions and Nearest Neighbors
Deep neural networks (DNNs) are notorious for their vulnerability to adversarial attacks, which are small perturbations added to their input images to mislead their prediction. Detection of adversarial examples is, there…
Robust classificationGompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by v…
image-classificationImage ClassificationInstance SegmentationLanguage Modeling+4CombU: A Combined Unit Activation for Fitting Mathematical Expressions with Neural Networks
The activation functions are fundamental to neural networks as they introduce non-linearity into data relationships, thereby enabling deep networks to approximate complex data relations. Existing efforts to enhance neura…