Evaluating Structural Generalization in Neural Machine Translation
Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures. Since it is regarded as a desired property of neural models, recent work has assessed compositional generalization in machine translation as well as semantic parsing. However, previous evaluations with machine translation have focused mostly on lexical generalization (i.e., generalization to unseen combinations of known words). Thus, it remains unclear to what extent models can translate sentences that require structural generalization (i.e., generalization to different sorts of syntactic structures). To address this question, we construct SGET, a machine translation dataset covering various types of compositional generalization with control of words and sentence structures. We evaluate neural machine translation models on SGET and show that they struggle more in structural generalization than in lexical generalization. We also find different performance trends in semantic parsing and machine translation, which indicates the importance of evaluations across various tasks.
Code (1)
Tasks
Machine TranslationSemantic ParsingSentenceTranslationSimilar Papers 제목 키워드 기반
Discrete Structural Planning for Generating Diverse Translations
Planning is important for humans when producing complex languages, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the global …
DiversityMachine TranslationSentenceText Generation+1Incorporating Structural Alignment Biases into an Attentional Neural Translation Model
Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive b…
DecoderMachine TranslationTranslationA Challenge Set Approach to Evaluating Machine Translation
Neural machine translation represents an exciting leap forward in translation quality. But what longstanding weaknesses does it resolve, and which remain? We address these questions with a challenge set approach to trans…
Machine TranslationTranslationOn Evaluating Multilingual Compositional Generalization with Translated Datasets
Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored. …
Machine TranslationSemantic ParsingTranslationStructural Supervision for Word Alignment and Machine Translation
Syntactic structure has long been argued to be potentially useful for enforcing accurate word alignment and improving generalization performance of machine translation. Unfortunately, existing wisdom demonstrates its sig…
DecoderMachine TranslationMulti-Task LearningTranslation+1