Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages
Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSegmentationTranslationSimilar Papers 제목 키워드 기반
An Evaluation of Subword Segmentation Strategies for Neural Machine Translation of Morphologically Rich Languages
Byte-Pair Encoding (BPE) (Sennrich et al., 2016) has become a standard pre-processing step when building neural machine translation systems. However, it is not clear whether this is an optimal strategy in all settings. W…
Language ModelingLanguage ModellingMachine TranslationTranslationMultitask Learning For Different Subword Segmentations In Neural Machine Translation
In Neural Machine Translation (NMT) the usage of subwords and characters as source and target units offers a simple and flexible solution for translation of rare and unseen words. However, selecting the optimal subword s…
DecoderMachine TranslationNMTSegmentation+1Bilingual Subword Segmentation for Neural Machine Translation
This paper proposed a new subword segmentation method for neural machine translation, {``}Bilingual Subword Segmentation,{''} which tokenizes sentences to minimize the difference between the number of subword units in a …
Machine TranslationSegmentationSentenceTranslationChallenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging
Word segmentation plays a pivotal role in improving any Arabic NLP application. Therefore, a lot of research has been spent in improving its accuracy. Off-the-shelf tools, however, are: i) complicated to use and ii) doma…
Machine TranslationPart-Of-Speech TaggingPOSPOS Tagging+3Modeling Multi-granularity Segmentation for Rare Words in Neural Machine Translation
Segmenting rare words into subwords has become a commonly used and effective way to alleviate the open vocabulary problem in Neural Machine Translation (NMT). The existing dominant segmentation methods either give rare w…
DiversityMachine TranslationNMTSegmentation+1