SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model
Transformers have revolutionized deep learning across various tasks, including audio representation learning, due to their powerful modeling capabilities. However, they often suffer from quadratic complexity in both GPU memory usage and computational inference time, affecting their efficiency. Recently, state space models (SSMs) like Mamba have emerged as a promising alternative, offering a more efficient approach by avoiding these complexities. Given these advantages, we explore the potential of SSM-based models in audio tasks. In this paper, we introduce Self-Supervised Audio Mamba (SSAMBA), the first self-supervised, attention-free, and SSM-based model for audio representation learning. SSAMBA leverages the bidirectional Mamba to capture complex audio patterns effectively. We incorporate a self-supervised pretraining framework that optimizes both discriminative and generative objectives, enabling the model to learn robust audio representations from large-scale, unlabeled datasets. We evaluated SSAMBA on various tasks such as audio classification, keyword spotting, and speaker identification. Our results demonstrate that SSAMBA outperforms the Self-Supervised Audio Spectrogram Transformer (SSAST) in most tasks. Notably, SSAMBA is approximately 92.7% faster in batch inference speed and 95.4% more memory-efficient than SSAST for the tiny model size with an input token size of 22k. These efficiency gains, combined with superior performance, underscore the effectiveness of SSAMBA's architectural innovation, making it a compelling choice for a wide range of audio processing applications.
Code (1)
Tasks
Audio ClassificationGPUKeyword SpottingMambaRepresentation LearningSpeaker IdentificationState Space ModelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
Despite its widespread adoption as the prominent neural architecture, the Transformer has spurred several independent lines of work to address its limitations. One such approach is selective state space models, which hav…
Language ModellingMambaState Space ModelsState Space Models for Bioacoustics: A Comparative Evaluation with Transformers
In this study, we evaluate the efficacy of the Mamba architecture bioacoustics by introducing BioMamba, a Mamba-based audio representation model for wildlife sounds. We pre-train a BioMamba using self-supervised learning…
Self-Supervised LearningAn overview of neural architectures for self-supervised audio representation learning from masked spectrograms
In recent years, self-supervised learning has amassed significant interest for training deep neural representations without labeled data. One such self-supervised learning approach is masked spectrogram modeling, where t…
Self-Supervised LearningRepresentation LearningAudio ClassificationAudio Mamba: Bidirectional State Space Model for Audio Representation Learning
Transformers have rapidly become the preferred choice for audio classification, surpassing methods based on CNNs. However, Audio Spectrogram Transformers (ASTs) exhibit quadratic scaling due to self-attention. The remova…
Audio ClassificationClassificationMambaRepresentation Learning+1Audio Mamba: Pretrained Audio State Space Model For Audio Tagging
Audio tagging is an important task of mapping audio samples to their corresponding categories. Recently endeavours that exploit transformer models in this field have achieved great success. However, the quadratic self-at…
Audio TaggingMambaState Space Models