One-layer transformers fail to solve the induction heads task
A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Selective Induction Heads: How Transformers Select Causal Structures In Context
Transformers have exhibited exceptional capabilities in sequence modeling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, attention circuits that enable copying tok…
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
In-context learning (ICL) is a hallmark capability of transformers, through which trained models learn to adapt to new tasks by leveraging information from the input context. Prior work has shown that ICL emerges in tran…
KV Shifting Attention Enhances Language Modeling
The current large language models are mainly based on decode-only structure transformers, which have great in-context learning (ICL) capabilities. It is generally believed that the important foundation of its ICL capabil…
In-Context LearningLanguage ModelingLanguage ModellingFrom Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers
Transformers can implement both generalizable algorithms (e.g., induction heads) and simple positional shortcuts (e.g., memorizing fixed output positions). In this work, we study how the choice of pretraining data distri…
On the Emergence of Induction Heads for In-Context Learning
Transformers have become the dominant architecture for natural language processing. Part of their success is owed to a remarkable capability known as in-context learning (ICL): they can acquire and apply novel associatio…