Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning

2024-04-13 · Yijiang Liu, Rongyu Zhang, Huanrui Yang, Kurt Keutzer, Yuan Du, Li Du, Shanghang Zhang

Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications, ranging from content generation to interactive entertainment, and artistic creation. However, the diversity of downstream tasks in multitask scenarios presents substantial adaptation challenges for LLMs. While traditional methods often succumb to knowledge confusion on their monolithic dense models, Mixture-of-Experts (MoE) has been emerged as a promising solution with its sparse architecture for effective task decoupling. Inspired by the principles of human cognitive neuroscience, we design a novel framework \texttt{Intuition-MoR1E} that leverages the inherent semantic clustering of instances to mimic the human brain to deal with multitask, offering implicit guidance to router for optimized feature allocation. Moreover, we introduce cutting-edge Rank-1 Experts formulation designed to manage a spectrum of intuitions, demonstrating enhanced parameter efficiency and effectiveness in multitask LLM finetuning. Extensive experiments demonstrate that Intuition-MoR1E achieves superior efficiency and 2.15\% overall accuracy improvement across 14 public datasets against other state-of-the-art baselines.

📄 PDF Abstract BibTeX arXiv:2404.08985

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityMixture-of-Experts

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

2023-12-01 · CVPR 2024 1 · Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang 외

Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. However, generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. Recent research sugges…

Chart Question Answering Document AI Image Captioning Mixture-of-Experts+2

TalkLoRA: Communication-Aware Mixture of Low-Rank Adaptation for Large Language Models

2026-04-07 · Lin Mu, Haiyang Wang, Li Ni, Lei Sang 외 arxiv

Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of Large Language Models (LLMs), and recent Mixture-of-Experts (MoE) extensions further enhance flexibility by dynamically combining multiple LoRA expert…

parameter-efficient fine-tuning

Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Experts

2025-05-20 · Xi Chen, Shen Yan, Juelin Zhu, Chen Chen 외

We introduce Land-MoE, a novel approach for multispectral land cover classification (MLCC). Spectral shift, which emerges from disparities in sensors and geospatial conditions, poses a significant challenge in this domai…

Domain Adaptation Domain Generalization Land Cover Classification Semantic Segmentation

Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs

2026-05-28 · Zhibo Zhang, Yuxi Li, Zhen Ouyang, Ling Shi 외 arxiv

Mixture-of-Experts (MoE) LLMs rely on sparse, router-driven expert activation, yet how safety alignment interacts with routed expert specialization remains underexplored. A common intuition is that safety behavior may be…

Autonomy-of-Experts Models

2025-01-22 · Ang Lv, Ruobing Xie, Yining Qian, Songhao Wu 외

Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's …

Decision Making Mixture-of-Experts