MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors

2024-05-29 · Renzhi Wang, Piji Li

Model editing aims to efficiently alter the behavior of Large Language Models (LLMs) within a desired scope, while ensuring no adverse impact on other inputs. Recent years have witnessed various model editing methods been proposed. However, these methods either exhibit poor overall performance or struggle to strike a balance between generalization and locality. We propose MEMoE, a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. MEMoE updates knowledge using a bypass MoE structure, keeping the original parameters unchanged to preserve the general ability of LLMs. And, the knowledge anchor routing ensures that inputs requiring similar knowledge are routed to the same expert, thereby enhancing the generalization of the updated knowledge. Experimental results show the superiority of our approach over both batch editing and sequential batch editing tasks, exhibiting exceptional overall performance alongside outstanding balance between generalization and locality. Our code will be available.

📄 PDF Abstract BibTeX arXiv:2405.19086

Code (0)

등록된 구현이 없습니다.

Tasks

Mixture-of-ExpertsModel Editing

Methods 이 논문이 사용한 방법론

Adapter 설명 없음

MoE 설명 없음

DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptation

2025-09-25 · Ziqi Chen, Gongyu Chen, Yihua Wang, Chaofan Ding 외 arxiv

Dialect speech embodies rich cultural and linguistic diversity, yet building text-to-speech (TTS) systems for dialects remains challenging due to scarce data, inconsistent orthographies, and complex phonetic variation. T…

Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE

2026-05-04 · Yangming Shi, Shixiang Zhu, Tao Shen, Zhimiao Yu 외 arxiv

We present Mamoda2.5, a unified AR-Diffusion framework that seamlessly integrates multimodal understanding and generation within a single architecture. To efficiently enhance the model's generation capability, we equip t…

Reinforcement Learning

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

2024-06-28 · Renzhi Wang, Piji Li

Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the developme…

Mixture-of-Experts Model Editing

Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

2026-05-15 · Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin, Daniil Medyakov 외 arxiv

Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE)…

knowledge editing

Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling

2026-03-05 · Yong Liu, Xingjian Su, Shiyu Wang, Haoran Zhang 외 arxiv

We introduce Timer-S1, a strong Mixture-of-Experts (MoE) time series foundation model with 8.3B total parameters, 0.75B activated parameters for each token, and a context length of 11.5K. To overcome the scalability bott…

Data Augmentation