Task Singular Vectors: Reducing Task Interference in Model Merging
Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.
Code (1)
Tasks
ClassificationImage ClassificationMulti-Label Image RecognitionTask ArithmeticSimilar Papers 제목 키워드 기반
Beyond Traditional Beamforming: Singular Vector Projection for MU-MIMO
This letter introduces a low-complexity beamforming approach for MU-MIMO systems with multiple data streams per user, minimizing inter-user interference and improving spectral efficiency (SE). The Interference-Optimized …
Computational EfficiencyManagementAdaRank: Adaptive Rank Pruning for Enhanced Model Merging
Model merging has emerged as a promising approach for unifying independently fine-tuned models into an integrated framework, significantly enhancing computational efficiency in multi-task learning. Recently, several SVD-…
Computational EfficiencymodelMulti-Task LearningRevisiting Weight Averaging for Model Merging
Model merging aims to build a multi-task learner by combining the parameters of individually fine-tuned models without additional training. While a straightforward approach is to average model parameters across tasks, th…
image-classificationImage ClassificationmodelMulti-Task LearningLoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging
While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its ability of improving models without requirin…
Language ModelingLanguage ModellingLarge Language ModelMultiLoRA: Democratizing LoRA for Better Multi-Task Learning
LoRA achieves remarkable resource efficiency and comparable performance when adapting LLMs for specific tasks. Since ChatGPT demonstrated superior performance on various tasks, there has been a growing desire to adapt on…
Multi-Task LearningNatural Language UnderstandingWorld Knowledge