PreCog: Exploring the Relation between Memorization and Performance in Pre-trained Language Models
Pre-trained Language Models such as BERT are impressive machines with the ability to memorize, possibly generalized learning examples. We present here a small, focused contribution to the analysis of the interplay between memorization and performance of BERT in downstream tasks. We propose PreCog, a measure for evaluating memorization from pre-training, and we analyze its correlation with the BERT's performance. Our experiments show that highly memorized examples are better classified, suggesting memorization is an essential key to success for BERT.
Code (0)
등록된 구현이 없습니다.
Tasks
MemorizationRelationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Memorization in Fine-tuned Language Models
Large language models (LLMs) have shown great capabilities in various tasks but also exhibited memorization of training data, raising tremendous privacy and copyright concerns. While prior works have studied memorization…
MemorizationMemorization in In-Context Learning
In-context learning (ICL) has proven to be an effective strategy for improving the performance of large language models (LLMs) with no additional training. However, the exact mechanism behind this performance improvement…
In-Context LearningMemorizationZero-Shot LearningTowards Differential Relational Privacy and its use in Question Answering
Memorization of the relation between entities in a dataset can lead to privacy issues when using a trained model for question answering. We introduce Relational Memorization (RM) to understand, quantify and control this …
MemorizationQuestion AnsweringLearning and Memorization
In the machine learning research community, it is generally believed that there is a tension between memorization and generalization. In this work we examine to what extent this tension exists by exploring if it is …
MemorizationThe Pitfalls of Memorization: When Memorization Hurts Generalization
Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these explanations.This behavior leads to poor generalization when the learned explanations …
Memorization