Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.
Code (2)
Tasks
Language ModelingLanguage ModellingMachine TranslationNMTSegmentationTranslationSimilar Papers 제목 키워드 기반
Adversarial Subword Regularization for Robust Neural Machine Translation
Exposing diverse subword segmentations to neural machine translation (NMT) models often improves the robustness of machine translation as NMT models can experience various subword candidates. However, the diversification…
Machine TranslationNMTTranslationSingle Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we use multiple segmentations in the trainin…
Machine TranslationSegmentationTranslationBPE-Dropout: Simple and Effective Subword Regularization
Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact whi…
Machine TranslationSegmentationTranslationMaxMatch-Dropout: Subword Regularization for WordPiece
We present a subword regularization method for WordPiece, which uses a maximum matching algorithm for tokenization. The proposed method, MaxMatch-Dropout, randomly drops words in a search using the maximum matching algor…
Machine TranslationText ClassificationTranslationSubword Regularization: An Analysis of Scalability and Generalization for End-to-End Automatic Speech Recognition
Subwords are the most widely used output units in end-to-end speech recognition. They combine the best of two worlds by modeling the majority of frequent words directly and at the same time allow open vocabulary speech r…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DiversityMachine Translation+4