Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout

2025-09-26 · Andi Zhang, Xuan Ding, Haofan Wang, Steven McDonagh, Samuel Kaski arxiv

We propose Orthogonal Monte Carlo Dropout, a mechanism that enforces strict orthogonality when combining sparse semantic vectors without extra time complexity. Low-Rank Adaptation (LoRA), a popular fine-tuning method for large models, typically trains a module to represent a specific concept such as an object or a style. When multiple LoRA modules are merged, for example to generate an object in a particular style, their outputs (semantic vectors) may interfere with each other. Our method guarantees that merged LoRA modules remain orthogonal and thus free from direct interference. However, empirical analysis reveals that such orthogonality does not lead to the semantic disentanglement highlighted in prior work on compositional adaptation. This finding suggests that inter-LoRA orthogonality alone may be insufficient for achieving true semantic compositionality, prompting a re-examination of its role in adapter merging.

📄 PDF Abstract BibTeX arXiv:2510.03262

Code (0)

등록된 구현이 없습니다.

PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

2026-06-09 · Gowtham Sivaramakrishnan, Sarvesha Kumar Kombaiah Seetha, Kishan Gupta Balaji, Santhosh Baradwaj Vaduvur Ranganathan arxiv

Access control in large language models (LLMs) requires modular mechanisms to enable domain-specific behavior without retraining or cross-domain interference. A common hypothesis is that interference during adapter compo…

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

2025-04-10 · Juzheng Zhang, Jiacheng You, Ashwinee Panda, Tom Goldstein

Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-ta…

Code Generation Continual Learning Mathematical Reasoning Natural Language Understanding+2

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models

2026-07-06 · Marian Lupascu, Sebastian Ripa, Mihai Trascau, Mariana-Iuliana Georgescu 외 arxiv

Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coh…

Tensorized Clustered LoRA Merging for Multi-Task Interference

2025-08-06 · Zhan Su, Fengran Mo, Guojun Liang, Jinghan Zhang 외 arxiv

Despite the success of the monolithic dense paradigm of large language models (LLMs), the LoRA adapters offer an efficient solution by fine-tuning small task-specific modules and merging them with the base model. However…

Question Answering

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

2026-06-02 · Zhengbao He, Ruiqi Ding, Zhehao Huang, Ruikai Yang 외 arxiv

Low-rank adaptation (LoRA) enables parameter-efficient specialization of foundation models, but the proliferation of task-specific adapters fragments capabilities across many adapters, complicating reuse and deployment. …