PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning
Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningGeneral KnowledgeMixture-of-ExpertsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget
Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) with conditionally-activated parallel experts. However, serving MoE models on memory-constrained devices is challenging …
Mixture-of-Expertsobject-detectionObject DetectionText SummarizationPatch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks
In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis, resulting in significant computation reduction. The recently proposed \underline{p}atch-level rou…
Mixture-of-ExpertsSPMoE: Generate Multiple Pattern-Aware Outputs with Sparse Pattern Mixture of Experts
Many generation tasks follow a one-to-many mapping relationship: each input could be associated with multiple outputs. Existing methods like Conditional Variational AutoEncoder(CVAE) employ a latent variable to model thi…
DiversityMixture-of-ExpertsParaphrase GenerationAdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference
Mixture-of-Experts (MoE) models are designed to enhance the efficiency of large language models (LLMs) without proportionally increasing the computational demands. However, their deployment on edge devices still faces si…
ManagementMixture-of-ExpertsSensitivityMLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts
Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token. While architecturally simple, this is computationally ineffic…