Length independent generalization bounds for deep SSM architectures
Many state-of-the-art models trained on long-range sequences, for example S4, S5 or LRU, are made of sequential blocks combining State-Space Models (SSMs) with neural networks. In this paper we provide a PAC bound that holds for these kind of architectures with stable SSM blocks and does not depend on the length of the input sequence. Imposing stability of the SSM blocks is a standard practice in the literature, and it is known to help performance. Our results provide a theoretical justification for the use of stable SSM blocks as the proposed PAC bound decreases as the degree of stability of the SSM blocks increases.
Code (0)
등록된 구현이 없습니다.
Tasks
Generalization BoundsState Space ModelsSimilar Papers 제목 키워드 기반
Sequence Length Independent Norm-Based Generalization Bounds for Transformers
This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel co…
Generalization BoundsOn Rank-Dependent Generalisation Error Bounds for Transformers
In this paper, we introduce various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms. These bounds are contingent on the rank of each class of matrices. …
Generalization Boundsf-divergences and their applications in lossy compression and bounding generalization error
In this paper, we provide three applications for $f$-divergences: (i) we introduce Sanov's upper bound on the tail probability of the sum of independent random variables based on super-modular $f$-divergence and show tha…
Generalization Bound for a General Class of Neural Ordinary Differential Equations
Neural ordinary differential equations (neural ODEs) are a popular type of deep learning model that operate with continuous-depth architectures. To assess how well such models perform on unseen data, it is crucial to und…
Length Generalization Bounds for Transformers
Length generalization is a key property of a learning algorithm that enables it to make correct predictions on inputs of any length, given finite training data. To provide such a guarantee, one needs to be able to comput…