Linear Complexity Self-Supervised Learning for Music Understanding with Random Quantizer
In recent years, foundation models have become very popular due to their exceptional performance, mainly in natural language (NLP) tasks where they were first introduced. These models usually consist of hundreds of millions, or even billions, of parameters, making them resource-intensive during training and in production systems, leading to increased costs. This paper focuses on the reduction of a foundation's model size when applied to music information retrieval (MIR) tasks. Our research combines the Branchformer architecture with SummaryMixing, which were first applied in speech recognition, along with a random quantization process. To facilitate reproducibility, we conduct pre-training on publicly available datasets, complemented by a proprietary dataset comparable in scale to other private datasets reported in the literature. We ensure robust evaluation by using a framework consisting of a variety of downstream MIR tasks. Our results show that our architecture achieves competitive performance when compared with other state-of-the-art models that use multi-head self-attention, while reducing the model size from 8.5% up to 12.3%.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningInformation RetrievalSpeech RecognitionSimilar Papers 제목 키워드 기반
MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization
Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detec…
Contrastive LearningKey DetectionMusic TaggingQuantization+2S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification
In this paper, we propose S3T, a self-supervised pre-training method with Swin Transformer for music classification, aiming to learn meaningful music representations from massive easily accessible unlabeled music data. S…
ClassificationData AugmentationGenre classificationMusic Classification+3MusiCoder: A Universal Music-Acoustic Encoder Based on Transformers
Music annotation has always been one of the critical topics in the field of Music Information Retrieval (MIR). Traditional models use supervised learning for music annotation tasks. However, as supervised machine learnin…
Genre classificationInformation RetrievalMusic Genre ClassificationMusic Information Retrieval+3MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training
Self-supervised learning (SSL) has recently emerged as a promising paradigm for training generalisable models on large-scale data in the fields of vision, text, and speech. Although SSL has been proven effective in speec…
Language ModellingQuantizationSelf-Supervised LearningContrastive Learning of Musical Representations
While deep learning has enabled great advances in many areas of music, labeled music datasets remain especially hard, expensive, and time-consuming to create. In this work, we introduce SimCLR to the music domain and con…
Contrastive LearningLinear evaluationMusic Auto-TaggingMusic Classification+1