Hierarchical Transformer for Multilingual Machine Translation

2021-03-05 · EACL (VarDial) 2021 4 · Albina Khusainova, Adil Khan, Adín Ramírez Rivera, Vitaly Romanov

The choice of parameter sharing strategy in multilingual machine translation models determines how optimally parameter space is used and hence, directly influences ultimate translation quality. Inspired by linguistic trees that show the degree of relatedness between different languages, the new general approach to parameter sharing in multilingual machine translation was suggested recently. The main idea is to use these expert language hierarchies as a basis for multilingual architecture: the closer two languages are, the more parameters they share. In this work, we test this idea using the Transformer architecture and show that despite the success in previous work there are problems inherent to training such hierarchical models. We demonstrate that in case of carefully chosen training strategy the hierarchical architecture can outperform bilingual models and multilingual models with full parameter sharing.

📄 PDF Abstract BibTeX arXiv:2103.03589

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation Translation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Residual Connection 설명 없음

Adam 설명 없음

Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation

2025-09-22 · Yiwen Guan, Jacob Whitehill arxiv

Multilingual translation suffers from computational redundancy, especially when translating into multiple languages simultaneously. In addition, translation quality can suffer for low-resource languages. To address this,…

Machine Translation Speech Recognition

A Framework for Hierarchical Multilingual Machine Translation

2020-05-12 · Ion Madrazo Azpiazu, Maria Soledad Pera

Multilingual machine translation has recently been in vogue given its potential for improving machine translation performance for low-resource languages via transfer learning. Empirical examinations demonstrating the suc…

Machine Translation Transfer Learning Translation

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders

2020-12-31 · Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi 외

Multilingual machine translation enables a single model to translate between different languages. Most existing multilingual machine translation systems adopt a randomly initialized Transformer backbone. In this work, in…

Language Modeling Language Modelling Machine Translation Translation+1

Non-Autoregressive Models are Better Multilingual Translators

2021-09-29 · ICLR 2022 4 · Zhenqiao Song, Hao Zhou, Lihua Qian, Jingjing Xu 외

Multilingual machine translation aims to develop a single model for multiple language directions. However, existing multilingual models based on Transformer are limited in terms of both translation performance and infere…

Decoder Machine Translation Sentence Translation

Multilingual Non-Autoregressive Machine Translation without Knowledge Distillation

2025-02-06 · Chenyang Huang, Fei Huang, Zaixiang Zheng, Osmar R. Zaïane 외

Multilingual neural machine translation (MNMT) aims at using one single model for multiple translation directions. Recent work applies non-autoregressive Transformers to improve the efficiency of MNMT, but requires expen…

Knowledge Distillation Machine Translation Translation