Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performance. When allocating smoothed probability, original label smoothing treats the source-side words that would never appear in the target language equally to the real target-side words, which could bias the translation model. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation from both translation quality and model's calibration. Our code is released at https://github.com/PKUnlp-icler/MLS
Code (2)
Tasks
Machine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Focus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal pe…
Machine TranslationTranslationFocus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal pe…
Machine TranslationTranslationFocus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performa…
Machine TranslationTranslationText Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Before entering the neural network, a token needs to be converted to its one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained…
Data AugmentationLanguage ModelingLanguage ModellingSentence+2Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Before entering the neural network, a token is generally converted to the corresponding one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate …
Data AugmentationLanguage ModelingLanguage ModellingSentence+2