LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks

2024-10-16 · Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, Samy Jelassi

Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of Large Language Models (LLMs). We study how different LoRA modules can be merged to achieve skill composition -- testing the performance of the merged model on a target task that involves combining multiple skills, each skill coming from a single LoRA. This setup is favorable when it is difficult to obtain training data for the target task and when it can be decomposed into multiple skills. First, we identify practically occurring use-cases that can be studied under the realm of skill composition, e.g. solving hard math-word problems with code, creating a bot to answer questions on proprietary manuals or about domain-specialized corpora. Our main contribution is to show that concatenation of LoRAs (CAT), which optimally weights LoRAs that were individually trained on different skills, outperforms existing model- and data- merging techniques; for instance on math-word problems, CAT beats these methods by an average of 43% and 12% respectively. Thus, this paper advocates model merging as an efficient way to solve compositional tasks and underscores CAT as a simple, compute-friendly and effective procedure. To our knowledge, this is the first work demonstrating the superiority of model merging over data mixing for binary skill composition tasks. Code and data are available at https://github.com/aksh555/LoRA-Soups

📄 PDF Abstract BibTeX arXiv:2410.13025

Code (1)

aksh555/lora-soups 공식 구현 pytorch

Tasks

Mathparameter-efficient fine-tuning

The Appeal and Reality of Recycling LoRAs with Adaptive Merging

2026-02-12 · Haokun Liu, Gyung Hyun Je, Marco Ciccone, Zhenlin Xu 외 arxiv

The widespread availability of fine-tuned LoRA modules for open pre-trained models has led to an interest in methods that can adaptively merge LoRAs to improve performance. These methods typically include some way of sel…

K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

2025-10-15 · Donald Shenaj, Ondrej Bohdal, Taha Ceritli, Mete Ozay 외 arxiv

On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile …

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

2024-09-24 · Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li 외

Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modu…

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

2024-02-18 · Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han 외

LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional modules represent diverse skills. Combining existing LoRAs to address new …

Math

Mixture of LoRA Experts

2024-04-21 · Xun Wu, Shaohan Huang, Furu Wei

LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as o…