MaLoRA: Gated Modality LoRA for Key-Space Alignment in Multimodal LLM Fine-Tuning

2025-10-30 · Xinhan Zheng, Huyu Wu, Xueting Wang, Duo Su, Haiyun Jiang arxiv

Multimodal large language models (MLLMs) exhibit a pronounced preference for textual inputs when processing vision-language data, limiting their ability to reason effectively from visual evidence. Unlike prior studies that attribute this text bias to external factors such as data imbalance or instruction tuning, we propose that the bias originates from the model's internal architecture. Specifically, we hypothesize that visual key vectors (Visual Keys) are out-of-distribution (OOD) relative to the text key space learned during language-only pretraining. Consequently, these visual keys receive systematically lower similarity scores during attention computation, leading to their under-utilization in the context representation. To validate this hypothesis, we extract key vectors from LLaVA and Qwen2.5-VL and analyze their distributional structures using qualitative (t-SNE) and quantitative (Jensen-Shannon divergence) methods. The results provide direct evidence that visual and textual keys occupy markedly distinct subspaces within the attention space. The inter-modal divergence is statistically significant, exceeding intra-modal variation by several orders of magnitude. These findings reveal that text bias arises from an intrinsic misalignment within the attention key space rather than solely from external data factors.

📄 PDF Abstract BibTeX arXiv:2510.26721

Code (0)

등록된 구현이 없습니다.

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

2024-10-30 · Xujia Wang, Haiyan Zhao, Shuo Wang, Hanqing Wang 외

Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have significantly improved the adaptation of LLMs to downstream tasks in a resource-efficient manner. However, in multi-task scenarios, challenges such as trainin…

Computational Efficiency Mixture-of-Experts Multi-Task Learning parameter-efficient fine-tuning

Multiagent Reinforcement Learning with an Attention Mechanism for Improving Energy Efficiency in LoRa Networks

2023-09-16 · Xu Zhang, Ziqi Lin, Shimin Gong, Bo Gu 외

Long Range (LoRa) wireless technology, characterized by low power consumption and a long communication range, is regarded as one of the enabling technologies for the Industrial Internet of Things (IIoT). However, as the …

Closing the gap in multimodal medical representation alignment

2026-02-23 · Eleonora Grassucci, Giordano Cicchetti, Danilo Comminiello arxiv

In multimodal learning, CLIP has emerged as the de-facto approach for mapping different modalities into a shared latent space by bringing semantically similar representations closer while pushing apart dissimilar ones. H…

Cross-Modal Retrieval Image Captioning

Selective State-Space Adaptation and Retrieval for Language Model Reasoning

2026-07-21 · Atahan Dokme, Larry Heck arxiv

Low-rank adaptation introduces a static learned update applied identically to every input. The update provides task-level adaptation but does not explicitly represent token-level or instance-level state variation. A fami…

Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP

2024-06-25 · Sedigheh Eslami, Gerard de Melo

Contrastive Language--Image Pre-training (CLIP) has manifested remarkable improvements in zero-shot classification and cross-modal vision-language tasks. Yet, from a geometrical point of view, the CLIP embedding space ha…

cross-modal alignment Image Classification text similarity zero-shot-classification+2