MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMs
Knowledge editing (KE) enables precise modifications to factual content in large language models (LLMs). Existing KE methods are largely designed for dense architectures, limiting their applicability to the increasingly prevalent sparse Mixture-of-Experts (MoE) models that underpin modern scalable LLMs. Although MoEs offer strong efficiency and capacity scaling, naively adapting dense-model editors is both computationally costly and prone to routing distribution shifts that undermine stability and consistency. To address these challenges, we introduce MoEEdit, the first routing-stable framework for parameter-modifying knowledge editing in MoE LLMs. Our method reparameterizes expert updates via per-expert null-space projections that keep router inputs invariant and thereby suppress routing shifts. The resulting block-structured optimization is solved efficiently with a block coordinate descent (BCD) solver. Experiments show that MoEEdit attains state-of-the-art efficacy and generalization while preserving high specificity and routing stability, with superior compute and memory efficiency. These results establish a robust foundation for scalable, precise knowledge editing in sparse LLMs and underscore the importance of routing-stable interventions.
Code (0)
등록된 구현이 없습니다.
Tasks
knowledge editingSimilar Papers 제목 키워드 기반
MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors
Model editing aims to efficiently alter the behavior of Large Language Models (LLMs) within a desired scope, while ensuring no adverse impact on other inputs. Recent years have witnessed various model editing methods bee…
Mixture-of-ExpertsModel EditingLEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the developme…
Mixture-of-ExpertsModel EditingStable Routing for Mixture-of-Experts in Class-Incremental Learning
Class-incremental learning (CIL) requires models to learn new classes sequentially while preserving prior knowledge. Recently, approaches that combine pre-trained models with mixture-of-experts (MoE) have received increa…
class-incremental learningSpectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures
Mixture of Experts (MoE) architectures enable efficient scaling of neural networks but suffer from expert collapse, where routing converges to a few dominant experts. This reduces model capacity and causes catastrophic i…
UniAdapt: A Universal Adapter for Knowledge Calibration
Large Language Models (LLMs) require frequent updates to correct errors and keep pace with continuously evolving knowledge in a timely and effective manner. Recent research in it model editing has highlighted the challen…
Mixture-of-ExpertsModel EditingRetrieval-augmented GenerationSemantic Similarity+1