Meaningless yet meaningful: Morphology grounded subword-level NMT
We explore the use of two independent subsystems Byte Pair Encoding (BPE) and Morfessor as basic units for subword-level neural machine translation (NMT). We show that, for linguistically distant language-pairs Morfessor-based segmentation algorithm produces significantly better quality translation than BPE. However, for close language-pairs BPE-based subword-NMT may translate better than Morfessor-based subword-NMT. We propose a combined approach of these two segmentation algorithms Morfessor-BPE (M-BPE) which outperforms these two baseline systems in terms of BLEU score. Our results are supported by experiments on three language-pairs: English-Hindi, Bengali-Hindi and English-Bengali.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTSegmentationTranslationTransliterationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MorphTok: Morphologically Grounded Tokenization for Indian Languages
Tokenization is a crucial step in NLP, especially with the rise of large language models (LLMs), impacting downstream performance, computational cost, and efficiency. Existing LLMs rely on the classical Byte-pair Encodin…
Language ModelingLanguage ModellingMachine TranslationSegmentation+1Subword-level Composition Functions for Learning Word Embeddings
Subword-level information is crucial for capturing the meaning and morphology of words, especially for out-of-vocabulary entries. We propose CNN- and RNN-based subword-level composition functions for learning word embedd…
Learning Word EmbeddingsWord EmbeddingsOptimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fideli…
Dependency ParsingSentiment AnalysisMorphology Informed Selections for Subword Vocabulary Size
Currently, guidance around selection of an optimal or appropriate subword vocabulary size is incomplete and confusing at best. Using a measure of subword-morpheme overlap, our analysis shows that one can find a "sweet sp…
How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?
Data-driven subword segmentation has become the default strategy for open-vocabulary machine translation and other NLP tasks, but may not be sufficiently generic for optimal learning of non-concatenative morphology. We d…
Machine TranslationSegmentationTranslation