On Compositional Generalization of Neural Machine Translation
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.
Code (1)
Tasks
Domain GeneralizationMachine TranslationNMTSentenceTranslationSimilar Papers 제목 키워드 기반
Evaluating Structural Generalization in Neural Machine Translation
Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures. Since it is regarded as a desired property of neural models, recent work has a…
Machine TranslationSemantic ParsingSentenceTranslationOn Evaluating Multilingual Compositional Generalization with Translated Datasets
Compositional generalization allows efficient learning and human-like inductive biases. Since most research investigating compositional generalization in NLP is done on English, important questions remain underexplored. …
Machine TranslationSemantic ParsingTranslationCompositional Generalization by Factorizing Alignment and Translation
Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, hum…
Machine TranslationSystematic GeneralizationTranslationCompositional Generalization via Neural-Symbolic Stack Machines
Despite achieving tremendous success, existing deep learning models have exposed limitations in compositional generalization, the capability to learn compositional rules and apply them to unseen cases in a systematic man…
Few-Shot LearningMachine TranslationTranslationCan Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN
Despite their practical success, modern seq2seq architectures are unable to generalize systematically on several SCAN tasks. Hence, it is not clear if SCAN-style compositional generalization is useful in realistic NLP ta…
Machine TranslationTranslation