Multi-Format Contrastive Learning of Audio Representations
Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio ClassificationContrastive LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CALM: Contrastive Aligned Audio-Language Multirate and Multimodal Representations
Deriving multimodal representations of audio and lexical inputs is a central problem in Natural Language Understanding (NLU). In this paper, we present Contrastive Aligned Audio-Language Multirate and Multimodal Represen…
Emotion RecognitionNatural Language UnderstandingEnriched Music Representations with Multiple Cross-modal Contrastive Learning
Modeling various aspects that make a music piece unique is a challenging task, requiring the combination of multiple sources of information. Deep learning is commonly used to obtain representations using various sources …
Contrastive LearningGenre classificationJoint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization
We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented …
FLAP: Fast Language-Audio Pre-training
We propose Fast Language-Audio Pre-training (FLAP), a self-supervised approach that efficiently and effectively learns aligned audio and language representations through masking, contrastive learning and reconstruction. …
AudioCapsContrastive LearningRetrievalText RetrievalContrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learning either global representations useful for high-level underst…
ClassificationDeepFake DetectionFace SwappingGeneral Classification+4