MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages, enabling broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Adaptive MultiScale-Headed Attention (Ada-MSHA), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and improves the potential to discover a better strategy than previous methods. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. Our code is available at https://github.com/ictnlp/MoCE.
Code (1)
Tasks
Machine TranslationSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts
Electroencephalography (EEG)-based multimodal learning integrates brain signals with complementary modalities to improve mental state assessment, providing great clinical potential. The effectiveness of such paradigms la…
Representation LearningEmotion RecognitionTask-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts
Masked Autoencoder~(MAE) is a prevailing self-supervised learning method that achieves promising results in model pre-training. However, when the various downstream tasks have data distributions different from the pre-tr…
Mixture-of-ExpertsSelf-Supervised LearningMixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specializ…
Enhancing Molecular Property Prediction via Mixture of Collaborative Experts
Molecular Property Prediction (MPP) task involves predicting biochemical properties based on molecular features, such as molecular graph structures, contributing to the discovery of lead compounds in drug development. To…
Decision MakingDiversityMolecular Property PredictionPrediction+1Complexity Experts are Task-Discriminative Learners for Any Image Restoration
Recent advancements in all-in-one image restoration models have revolutionized the ability to address diverse degradations through a unified framework. However, parameters tied to specific tasks often remain inactive for…
AttributeBlind All-in-One Image RestorationImage RestorationMixture-of-Experts