Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal performance, since the soft label produced by label smoothing still considers the source-side words that would not appear at the target side. To address this issue, we propose Masked Label Smoothing (MLS), a new mechanism that masks the soft label probability of source-side words to zero. Simple yet effective, MLS manages to better integrate label smoothing with vocabulary sharing and hence improves the quality of the translation. Our extensive experiments show that MLS consistently yields improvement over original label smoothing on different datasets, including bilingual and multilingual translation in both BLEU and calibration scores.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Focus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that jointly adopting these two techniques can be conflicting and even leads to sub-optimal pe…
Machine TranslationTranslationFocus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performa…
Machine TranslationTranslationFocus on the Target’s Vocabulary: Masked Label Smoothing for Machine Translation
Label smoothing and vocabulary sharing are two widely used techniques in neural machine translation models. However, we argue that simply applying both techniques can be conflicting and even leads to sub-optimal performa…
Machine TranslationTranslationText Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Before entering the neural network, a token needs to be converted to its one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained…
Data AugmentationLanguage ModelingLanguage ModellingSentence+2Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
Before entering the neural network, a token is generally converted to the corresponding one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate …
Data AugmentationLanguage ModelingLanguage ModellingSentence+2