paper-with-me

홈 › Papers

Sequence Length Independent Norm-Based Generalization Bounds for Transformers

2023-10-19 · Jacob Trauger, Ambuj Tewari

This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel covering number bounds for the function class of bounded linear transformations to upper bound the Rademacher complexity of the Transformer. Furthermore, we show this generalization bound applies to the common Transformer training technique of masking and then predicting the masked word. We also run a simulated study on a sparse majority data set that empirically validates our theoretical findings.

📄 PDF Abstract BibTeX arXiv:2310.13088

Code (1)

traugerjacob/transformer-gen-bounds 공식 구현 tf

Tasks

Generalization Bounds

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically
Multi-Head Attention 설명 없음
Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Position-Wise Feed-Forward Layer 설명 없음
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…
Residual Connection 설명 없음

Similar Papers 제목 키워드 기반

On Rank-Dependent Generalisation Error Bounds for Transformers

2024-10-15 · Lan V. Truong

In this paper, we introduce various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms. These bounds are contingent on the rank of each class of matrices. …

Generalization Bounds

Effective Sample Size and Generalization Bounds for Temporal Networks

2025-08-08 · Barak Gahtan, Alex M. Bronstein arxiv

Learning from time series is fundamentally different from learning from i.i.d.\ data: temporal dependence can make long sequences effectively information-poor, yet standard evaluation protocols conflate sequence length w…

Length independent generalization bounds for deep SSM architectures

2024-05-30 · Dániel Rácz, Mihály Petreczky, Bálint Daróczy

Many state-of-the-art models trained on long-range sequences, for example S4, S5 or LRU, are made of sequential blocks combining State-Space Models (SSMs) with neural networks. In this paper we provide a PAC bound that h…

Generalization BoundsState Space Models

Quantitative Bounds for Length Generalization in Transformers

2025-10-30 · Zachary Izzo, Eshaan Nichani, Jason D. Lee arxiv

We study the problem of length generalization (LG) in transformers: the ability of a model trained on shorter sequences to maintain performance when evaluated on much longer, previously unseen inputs. Prior work by Huang…

Norm-based Generalization Bounds for Compositionally Sparse Neural Networks

2023-01-28 · Tomer Galanti, Mengjia Xu, Liane Galanti, Tomaso Poggio

In this paper, we investigate the Rademacher complexity of deep sparse neural networks, where each neuron receives a small number of inputs. We prove generalization bounds for multilayered sparse ReLU neural networks, in…

Generalization Bounds