Birth of a Transformer: A Memory Viewpoint
Large language models based on transformers have achieved great empirical successes. However, as they are deployed more widely, there is a growing need to better understand their internal mechanisms in order to make them more reliable. These models appear to store vast amounts of knowledge from their training data, and to adapt quickly to new information provided in their context or prompt. We study how transformers balance these two types of knowledge by considering a synthetic setup where tokens are generated from either global or context-specific bigram distributions. By a careful empirical analysis of the training process on a simplified two-layer transformer, we illustrate the fast learning of global bigrams and the slower development of an "induction head" mechanism for the in-context bigrams. We highlight the role of weight matrices as associative memories, provide theoretical insights on how gradients enable their learning during training, and study the role of data-distributional properties.
Code (1)
Similar Papers 제목 키워드 기반
Multi-view Attention for gestational age at birth prediction
We present our method for gestational age at birth prediction for the SLCN (surface learning for clinical neuroimaging) challenge. Our method is based on a multi-view shape analysis technique that captures 2D renderings …
PredictionregressionTransformer Based Multi-Target Bernoulli Tracking for Maritime Radar
Multi-target tracking in the maritime domain is a challenging problem due to the non-Gaussian and fluctuating characteristics of sea clutter. This article investigates the use of machine learning (ML) to the detection an…
An Integrated Optimization and Deep Learning Pipeline for Predicting Live Birth Success in IVF Using Feature Optimization and Transformer-Based Models
In vitro fertilization (IVF) is a widely utilized assisted reproductive technology, yet predicting its success remains challenging due to the multifaceted interplay of clinical, demographic, and procedural factors. This …
feature selectionCLaC at SMM4H 2020: Birth Defect Mention Detection
For the detection of personal tweets, where a parent speaks of a child’s birth defect, CLaC combines ELMo word embeddings and gazetteer lists from external resources with a GCNN (for encoding dependencies), in a multi la…
Word EmbeddingsBabyNet: Residual Transformer Module for Birth Weight Prediction on Fetal Ultrasound Video
Predicting fetal weight at birth is an important aspect of perinatal care, particularly in the context of antenatal management, which includes the planned timing and the mode of delivery. Accurate prediction of weight us…
Management