Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.
Code (5)
Tasks
Language ModelingLanguage ModellingMambaState Space ModelsSimilar Papers 제목 키워드 기반
Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers
A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers. This paper studies a unifying matrix mixer view of sequence mixers that…
MambaState Space ModelsRepeat After Me: Transformers are Better than State Space Models at Copying
Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized …
State Space ModelsState Space Models are Comparable to Transformers in Estimating Functions with Dynamic Smoothness
Deep neural networks based on state space models (SSMs) are attracting much attention in sequence modeling since their computational cost is significantly smaller than that of Transformers. While the capabilities of SSMs…
State Space ModelsFrom S4 to Mamba: A Comprehensive Survey on Structured State Space Models
Recent advancements in sequence modeling have led to the emergence of Structured State Space Models (SSMs) as an efficient alternative to Recurrent Neural Networks (RNNs) and Transformers, addressing challenges in long-r…
Computational EfficiencyMambaspeech-recognitionSpeech Recognition+3CodeSSM: Towards State Space Models for Code Understanding
Although transformers are widely used for various code-specific tasks, they have some significant limitations. In this paper, we investigate State Space Models (SSMs) as a potential alternative to transformers for code u…
Clone DetectionLanguage ModelingLanguage ModellingMasked Language Modeling+3