A Meta-Learning Perspective on Transformers for Causal Language Modeling

2023-10-09 · Xinbo Wu, Lav R. Varshney

The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process within the Transformer. Further, within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments in various settings.

📄 PDF Abstract BibTeX arXiv:2310.05884

Code (0)

등록된 구현이 없습니다.

Tasks

Causal Language ModelingLanguage ModelingLanguage Modelling Meta-Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Position-Wise Feed-Forward Layer 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Winner-Take-All Spiking Transformer for Language Modeling

2026-04-13 · Chenlin Zhou, Sihang Guo, Jiaqi Wang, Dongyang Ma 외 arxiv

Spiking Transformers, which combine the scalability of Transformers with the sparse, energy-efficient property of Spiking Neural Networks (SNNs), have achieved impressive results in neuromorphic and vision tasks and attr…

Natural Language Understanding

Minimizing Memorization in Meta-learning: A Causal Perspective

2021-09-29 · Yinjie Jiang, Zhengyu Chen, Luotian Yuan, Ying WEI 외

Meta-learning has emerged as a potent paradigm for quick learning of few-shot tasks, by leveraging the meta-knowledge learned from meta-training tasks. Well-generalized meta-knowledge that facilitates fast adaptation in …

Causal Inference Memorization Meta-Learning

Causal Evidence of Stack Representations in Modeling Counter Languages Using Transformers

2026-06-02 · Nishit Singh arxiv

Formal languages have proven to be effective conduits to understand the inner mechanisms of transformers. Past work has shown that transformers trained on next token prediction over counter languages learn representation…

How Transformers Learn Causal Structure with Gradient Descent

2024-02-22 · Eshaan Nichani, Alex Damian, Jason D. Lee

The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attentio…

In-Context Learning

On the Runway Cascade of Transformers for Language Modeling

2026-01-20 · Hunjae Lee, Corey Clark arxiv

In decoder-only (causal) transformers, the computation graph created by causal masking routes information through both direct-path attention and indirect paths formed by intermediate tokens. We denote these indirect path…

Information Retrieval