Memorisation versus Generalisation in Pre-trained Language Models
State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such as few-shot learning and rare entity recognition. To mitigate such limitations, we propose an extension based on prototypical networks that improves performance in low-resource named entity recognition tasks.
Code (1)
Tasks
Few-Shot LearningLow Resource Named Entity Recognitionnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks
Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, bu…
image-classificationImage ClassificationMemorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation
When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others. Yet, memorisation is not easily expressed as a binary feature that is good or bad: indiv…
counterfactualMachine TranslationNMTPositionWhat do larger image classifiers memorise?
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely rand…
image-classificationImage ClassificationKnowledge DistillationMemorization+1SoK: Memorisation in machine learning
Quantifying the impact of individual data samples on machine learning models is an open research problem. This is particularly relevant when complex and high-dimensional relationships have to be learned from a limited sa…
Do Smaller Language Models Answer Contextualised Questions Through Memorisation Or Generalisation?
A distinction is often drawn between a model's ability to predict a label for an evaluation sample that is directly memorised from highly similar training samples versus an ability to predict the label via some method of…
Question AnsweringSemantic SimilaritySemantic Textual Similarity