BPE beyond Word Boundary: How NOT to use Multi Word Expressions in Neural Machine Translation
BPE tokenization merges characters into longer tokens by finding frequently occurring contiguous patterns within the word boundary. An intuitive relaxation would be to extend a BPE vocabulary with multi-word expressions (MWEs): bigrams (in\_a), trigrams (out\_of\_the), and skip-grams (he . his). In the context of Neural Machine Translation (NMT), we replace the least frequent subword/whole-word tokens with the most frequent MWEs. We find that these modifications to BPE end up hurting the model, resulting in a net drop of BLEU and chrF scores across two language pairs. We observe that naively extending BPE beyond word boundaries results in incoherent tokens which are themselves better represented as individual words. Moreover, we find that Pointwise Mutual Information (PMI) instead of frequency finds better MWEs (e.g., New\_York, Statue\_of\_Liberty, neither . nor) which consistently improves translation performance.We release all code at https://github.com/pegasus-lynx/mwe-bpe.
Code (1)
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Beyond Words: Deep Learning for Multiword Expressions and Collocations
Deep learning has recently shown much promise for NLP applications. Traditionally, in most NLP approaches, documents or sentences are represented by a sparse bag-of-words representation. There is now a lot of work which …
Deep LearningInformation RetrievalMachine TranslationDedicated Language Resources for Interdisciplinary Research on Multiword Expressions: Best Thing since Sliced Bread
Multiword expressions such as idioms (beat about the bush), collocations (plastic surgery) and lexical bundles (in the middle of) are challenging for disciplines like Natural Language Processing (NLP), psycholinguistics …
Language AcquisitionAspect Specific Opinion Expression Extraction using Attention based LSTM-CRF Network
Opinion phrase extraction is one of the key tasks in fine-grained sentiment analysis. While opinion expressions could be generic subjective expressions, aspect specific opinion expressions contain both the aspect as well…
SentenceSentiment AnalysisUsing large language models to estimate features of multi-word expressions: Concreteness, valence, arousal
This study investigates the potential of large language models (LLMs) to provide accurate estimates of concreteness, valence and arousal for multi-word expressions. Unlike previous artificial intelligence (AI) methods, L…
A data-driven approach to verbal multiword expression detection. PARSEME Shared Task system description paper
{``}Multiword expressions{''} are groups of words acting as a morphologic, syntactic and semantic unit in linguistic analysis. Verbal multiword expressions represent the subgroup of multiword expressions, namely that in …
feature selectionLemmatizationSentence