Emergent properties with repeated examples
We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityMemorizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Emergence and Synchronization in Chaotic Oscillators and in the Human Cortical Network
When we look at the world around us, we see complex physical systems and emergent phenomena. Emergence occurs when a system is observed to have properties that its parts do not have on their own. These properties or beha…
Beehive scale-free emergent dynamics
It has been repeatedly reported that the collective dynamics of social insects exhibit universal emergent properties similar to other complex systems. In this note, we study a previously published data set in which the p…
Emergent predictability in microbial ecosystems
Microbial ecosystems carry out essential functions for global climate, human health, and industry. These complex communities exhibit a surprising amount of functionally relevant diversity at all levels of taxonomic resol…
Compositional properties of emergent languages in deep learning
Recent findings in multi-agent deep learning systems point towards the emergence of compositional languages. These claims are often made without exact analysis or testing of the language. In this work, we analyze the eme…
Deep LearningEmergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
Fine-tuning LLMs on narrow harmful datasets can induce Emergent Misalignment (EM), where models exhibit misaligned behavior far beyond the fine-tuning distribution. We argue that emergent misalignment can be better under…