Papers Task Arithmetic
“Task Arithmetic” 태그가 달린 논문 61편 · 필터 해제
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one. However, parameter conflicts between models leads to performance degradation in averaging. While model routing addre…
Task ArithmeticEfficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach
Task vectors, which are derived from the difference between pre-trained and fine-tuned model weights, enable flexible task adaptation and model merging through arithmetic operations such as addition and negation. However…
Model EditingNegationTask ArithmeticTask Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts
Multi-task model merging offers an efficient solution for integrating knowledge from multiple fine-tuned models, mitigating the significant computational and storage demands associated with multi-task training. As a key …
NavigateTask ArithmeticSoup to go: mitigating forgetting during continual learning with model averaging
In continual learning, where task data arrives in a sequence, fine-tuning on later tasks will often lead to performance degradation on earlier tasks. This is especially pronounced when these tasks come from diverse domai…
Continual LearningTask ArithmeticBADTV: Unveiling Backdoor Threats in Third-Party Task Vectors
Task arithmetic in large-scale pre-trained models enables agile adaptation to diverse downstream tasks without extensive retraining. By leveraging task vectors (TVs), users can perform modular updates through simple arit…
Backdoor AttackTask ArithmeticWhen Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach
Generalized Class Discovery (GCD) clusters base and novel classes in a target domain, using supervision from a source domain with only base classes. Current methods often falter with distribution shifts and typically…
Domain AdaptationDomain GeneralizationRepresentation LearningTask ArithmeticBias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach
The use of language models (LMs) has increased considerably in recent years, and the biases and stereotypes in training data that are reflected in the LM outputs are causing social problems. In this paper, inspired by th…
Language ModelingLanguage ModellingMasked Language ModelingTask ArithmeticTask Arithmetic Through The Lens Of One-Shot Federated Learning
Task Arithmetic is a model merging technique that enables the combination of multiple models' capabilities into a single model through simple arithmetic in the weight space, without the need for additional fine-tuning or…
Federated LearningMulti-Task LearningTask ArithmeticMulti-Task Model Merging via Adaptive Weight Disentanglement
Model merging has recently gained attention as an economical and scalable approach to incorporate task-specific weights from various tasks into a unified multi-task model. For example, in Task Arithmetic (TA), adding the…
DisentanglementmodelTask ArithmeticTask Singular Vectors: Reducing Task Interference in Model Merging
Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is sus…
ClassificationImage ClassificationMulti-Label Image RecognitionTask ArithmeticBeyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics
Pretrained models have revolutionized deep learning by enabling significant performance improvements across a wide range of tasks, leveraging large-scale, pre-learned knowledge representations. However, deploying these m…
Knowledge DistillationMulti-Task LearningQuantizationTask ArithmeticATM: Improving Model Merging by Alternating Tuning and Merging
Model merging has recently emerged as a cost-efficient paradigm for multi-task learning. Among current approaches, task arithmetic stands out for its simplicity and effectiveness. In this paper, we motivate the effective…
Federated LearningMulti-Task LearningTask ArithmeticEfficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. Recent research on task arithmetic-based MTL demonstrates that merging the parameters of independently fi…
Mixture-of-ExpertsMulti-Task LearningTask ArithmeticTransfer LearningThe Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse
Model merging aims to efficiently combine the weights of multiple expert models, each trained on a specific task, into a single multi-task model, with strong performance across all tasks. When applied to all but the last…
Linear Mode ConnectivityTask ArithmeticNegMerge: Consensual Weight Negation for Strong Machine Unlearning
Machine unlearning aims to selectively remove specific knowledge from a model. Current methods, such as task arithmetic, rely on fine-tuning models on the forget set, generating a task vector, and subtracting it from the…
image-classificationImage ClassificationMachine UnlearningNegation+1What Matters for Model Merging at Scale?
Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development…
modelTask ArithmeticZero-shot GeneralizationTowards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration
Federated Learning has emerged as a promising paradigm for collaborative machine learning, while preserving user data privacy. Despite its potential, standard FL lacks support for diverse heterogeneous device prototypes,…
Federated LearningKnowledge DistillationTask ArithmeticTransfer LearningTask Arithmetic for Language Expansion in Speech Translation
Recent advances in large language models (LLMs) have gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-based speech translation (ST). However, expanding language pai…
Machine TranslationTask ArithmeticTranslationLocalize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic
Model merging offers an effective strategy to combine the strengths of multiple finetuned models into a unified model that preserves the specialized capabilities of each. Existing methods merge models in a global manner,…
Model CompressionTask ArithmeticFine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic
In recent years, task arithmetic has garnered increasing attention. This approach edits pre-trained models directly in weight space by combining the fine-tuned weights of various tasks into a unified model. Its efficienc…
ClassificationDisentanglementMulti-Task LearningTask Arithmetic