Routing in Sparsely-gated Language Models responds to Context
Language Models (LMs) recently incorporate mixture-of-experts layers consisting of a router and a collection of experts to scale up their parameter count given a fixed computational budget. Building on previous efforts indicating that token-expert assignments are predominantly influenced by token identities and positions, we trace routing decisions of similarity-annotated text pairs to evaluate the context sensitivity of learned token-expert assignments. We observe that routing in encoder layers mainly depends on (semantic) associations, but contextual cues provide an additional layer of refinement. Conversely, routing in decoder layers is more variable and markedly less sensitive to context.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderMixture-of-ExpertsSensitivitySimilar Papers 제목 키워드 기반
Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderMixture-of-Experts+2RANGER: Sparsely-Gated Mixture-of-Experts with Adaptive Retrieval Re-ranking for Pathology Report Generation
Pathology report generation remains a relatively under-explored downstream task, primarily due to the gigapixel scale and complex morphological heterogeneity of Whole Slide Images (WSIs). Existing pathology report genera…
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit.…
Machine TranslationMixture-of-ExpertsTranslationMoE-Based Learned Inertial Odometry for Bicycle Localization
GNSS suffers from multipath errors in urban canyons, making reliable bicycle localization difficult. Hand-crafted inertial alternatives, such as cycling dead reckoning and nonholonomic constraints, fail to generalize acr…
CoFi-PGMA: Counterfactual Policy Gradients under Filtered Feedback for Multi-Agent LLMs
Large language model (LLM) deployments increasingly rely on multi-agent architectures in which multiple models either compete through routing mechanisms or collaborate to produce a final answer. In both settings, the lea…