Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation
When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others. Yet, memorisation is not easily expressed as a binary feature that is good or bad: individual datapoints lie on a memorisation-generalisation continuum. What determines a datapoint's position on that spectrum, and how does that spectrum influence neural models' performance? We address these two questions for neural machine translation (NMT) models. We use the counterfactual memorisation metric to (1) build a resource that places 5M NMT datapoints on a memorisation-generalisation map, (2) illustrate how the datapoints' surface-level characteristics and a models' per-datum training signals are predictive of memorisation in NMT, (3) and describe the influence that subsets of that map have on NMT systems' performance.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualMachine TranslationNMTPositionSimilar Papers 제목 키워드 기반
Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks
Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, bu…
image-classificationImage ClassificationWhat do larger image classifiers memorise?
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely rand…
image-classificationImage ClassificationKnowledge DistillationMemorization+1SoK: Memorisation in machine learning
Quantifying the impact of individual data samples on machine learning models is an open research problem. This is particularly relevant when complex and high-dimensional relationships have to be learned from a limited sa…
Reducing Memorisation in Generative Models via Riemannian Bayesian Inference
Modern generative models can produce realistic samples, however, balancing memorisation and generalisation remains an open problem. We approach this challenge from a Bayesian perspective by focusing on the parameter spac…
Bayesian InferenceMemorisation versus Generalisation in Pre-trained Language Models
State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisati…
Few-Shot LearningLow Resource Named Entity Recognitionnamed-entity-recognitionNamed Entity Recognition+1