TensLoRA: Tensor Alternatives for Low-Rank Adaptation
Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have considered joint, tensor-based adaptations, but only in limited forms and without a systematic framework. We introduce TensLoRA, a unified framework that aggregates LoRA updates into higher-order tensors and models a broad family of tensor-based low-rank adaptations. Our formulation generalizes existing tensor-based methods and enables mode-specific compression rates, allowing parameter budgets to be tailored according to the modality and task. Experiments on vision and language benchmarks reveal that the tensor construction directly impacts performance, sometimes better than standard LoRA under similar parameter counts.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A study of the Multicriteria decision analysis based on the time-series features and a TOPSIS method proposal for a tensorial approach
A number of Multiple Criteria Decision Analysis (MCDA) methods have been developed to rank alternatives based on several decision criteria. Usually, MCDA methods deal with the criteria value at the time the decision is m…
Decision MakingTime SeriesTime Series AnalysisAFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Models
We present a novel Parameter-Efficient Fine-Tuning (PEFT) method, dubbed as Adaptive Freezing of Low Rank Adaptation (AFLoRA). Specifically, for each pre-trained frozen weight tensor, we add a parallel path of trainable …
parameter-efficient fine-tuningLoTR: Low Tensor Rank Weight Adaptation
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factor…
parameter-efficient fine-tuningTensor DecompositionMulticriteria decision support employing adaptive prediction in a tensor-based feature representation
Multicriteria decision analysis (MCDA) is a widely used tool to support decisions in which a set of alternatives should be ranked or classified based on multiple criteria. Recent studies in MCDA have shown the relevance …
Decision MakingPredictionTime SeriesLow-Rank Tensor Completion: A Pseudo-Bayesian Learning Approach
Low rank tensor completion, which solves a linear inverse problem with the principle of parsimony, is a powerful technique used in many application domains in computer vision and pattern recognition. As a surrogate funct…