Optimizing Segmentation Granularity for Neural Machine Translation
In neural machine translation (NMT), it is has become standard to translate using subword units to allow for an open vocabulary and improve accuracy on infrequent words. Byte-pair encoding (BPE) and its variants are the predominant approach to generating these subwords, as they are unsupervised, resource-free, and empirically effective. However, the granularity of these subword units is a hyperparameter to be tuned for each language and task, using methods such as grid search. Tuning may be done inexhaustively or skipped entirely due to resource constraints, leading to sub-optimal performance. In this paper, we propose a method to automatically tune this parameter using only one training pass. We incrementally introduce new vocabulary online based on the held-out validation loss, beginning with smaller, general subwords and adding larger, more specific units over the course of training. Our method matches the results found with grid search, optimizing segmentation granularity without any additional training time. We also show benefits in training efficiency and performance improvements for rare words due to the way embeddings for larger units are incrementally constructed by combining those from smaller units.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Improving Patent Translation using Bilingual Term Extraction and Re-tokenization for Chinese--Japanese
Unlike European languages, many Asian languages like Chinese and Japanese do not have typographic boundaries in written system. Word segmentation (tokenization) that break sentences down into individual words (tokens) is…
Chinese Word SegmentationMachine TranslationSegmentationTerm Extraction+1Modeling Multi-granularity Segmentation for Rare Words in Neural Machine Translation
Segmenting rare words into subwords has become a commonly used and effective way to alleviate the open vocabulary problem in Neural Machine Translation (NMT). The existing dominant segmentation methods either give rare w…
DiversityMachine TranslationNMTSegmentation+1Knowledge Graph Enhanced Neural Machine Translation via Multi-task Learning on Sub-entity Granularity
Previous studies combining knowledge graph (KG) with neural machine translation (NMT) have two problems: i) Knowledge under-utilization: they only focus on the entities that appear in both KG and training sentence pairs,…
Machine TranslationMulti-Task LearningNMTSentence+1Learning to Segment Inputs for NMT Favors Character-Level Processing
Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths, hence the modeling depth, and source and target voc…
Machine TranslationNavigateNMTSegmentation+1Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention
Natural language sentences, being hierarchical, can be represented at different levels of granularity, like words, subwords, or characters. But most neural machine translation systems require the sentence to be represent…
DecoderMachine TranslationSentenceTranslation