Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

2026-04-18 · Yixuan Tang, Yi Yang arxiv

Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update $ΔW = BA$ as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix $B$. Across tasks, $B$ repeatedly uses a small set of shared directions, while $A$ remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.

📄 PDF Abstract BibTeX arXiv:2604.16826

Code (0)

등록된 구현이 없습니다.

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

2026-02-27 · Lingfeng He, De Cheng, Huaijie Wang, Xi Yang 외 arxiv

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gaine…

parameter-efficient fine-tuning Continual Learning

SDFLoRA: Selective Decoupled Federated LoRA for Privacy-preserving Fine-tuning with Heterogeneous Clients

2026-01-16 · Zhikang Shen, Jianrong Lu, Haiyuan Wan, Jianhai Chen arxiv

Federated learning (FL) for large language models (LLMs) has attracted increasing attention as a privacy-preserving approach for adapting models over distributed data, where parameter-efficient methods such as Low-Rank A…

Federated Learning

Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

2026-08-27 · Yibo Feng arxiv

Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and rep…

class-incremental learning Continual Learning

NP-LoRA: Null Space Projection for Subject-Style LoRA Fusion

2025-11-14 · Chuheng Chen, Xiaofei Zhou, Geyuan Zhang, Yong Huang arxiv

Low-Rank Adaptation (LoRA) fusion enables the composition of subject and style representations for controllable generation without retraining. However, existing approaches primarily operate through weight-level merging, …

Crowded Scene Analysis: A Survey

2015-02-06 · Teng Li, Huan Chang, Meng Wang, Bingbing Ni 외

Automated scene analysis has been a topic of great interest in computer vision and cognitive science. Recently, with the growth of crowd phenomena in the real world, crowded scene analysis has attracted much attention. H…

Anomaly Detection Survey