Zero-Shot Audio Classification via Semantic Embeddings
In this paper, we study zero-shot learning in audio classification via semantic embeddings extracted from textual labels and sentence descriptions of sound classes. Our goal is to obtain a classifier that is capable of recognizing audio instances of sound classes that have no available training samples, but only semantic side information. We employ a bilinear compatibility framework to learn an acoustic-semantic projection between intermediate-level representations of audio instances and sound classes, i.e., acoustic embeddings and semantic embeddings. We use VGGish to extract deep acoustic embeddings from audio clips, and pre-trained language models (Word2Vec, GloVe, BERT) to generate either label embeddings from textual labels or sentence embeddings from sentence descriptions of sound classes. Audio classification is performed by a linear compatibility function that measures how compatible an acoustic embedding and a semantic embedding are. We evaluate the proposed method on a small balanced dataset ESC-50 and a large-scale unbalanced audio subset of AudioSet. The experimental results show that classification performance is significantly improved by involving sound classes that are semantically close to the test classes in training. Meanwhile, we demonstrate that both label embeddings and sentence embeddings are useful for zero-shot learning. Classification performance is improved by concatenating label/sentence embeddings generated with different language models. With their hybrid concatenations, the results are improved further.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio ClassificationClassificationGeneral ClassificationSentenceSentence EmbeddingsZero-shot Audio ClassificationZero-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Zero-Shot Audio Classification with Factored Linear and Nonlinear Acoustic-Semantic Projections
In this paper, we study zero-shot learning in audio classification through factored linear and nonlinear acoustic-semantic projections between audio instances and sound classes. Zero-shot learning in audio classification…
Audio ClassificationClassificationGeneral ClassificationZero-shot Audio Classification+1Zero-Shot Audio Classification using Image Embeddings
Supervised learning methods can solve the given problem in the presence of a large set of labeled data. However, the acquisition of a dataset covering all the target classes typically requires manual labeling which is ex…
Audio ClassificationClassificationZero-shot Audio ClassificationZero-Shot LearningZero-Shot Audio Classification Based on Class Label Embeddings
This paper proposes a zero-shot learning approach for audio classification based on the textual information about class labels without any audio samples from target classes. We propose an audio classification system buil…
Audio ClassificationClassificationGeneral ClassificationZero-shot Audio Classification+1Improving Audio Classification by Transitioning from Zero- to Few-Shot
State-of-the-art audio classification often employs a zero-shot approach, which involves comparing audio embeddings with embeddings from text describing the respective audio class. These embeddings are usually generated …
Contrastive LearningAudio ClassificationAVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings
In this paper, we propose a novel approach for generalized zero-shot learning in a multi-modal setting, where we have novel classes of audio/video during testing that are not seen during training. We use the semantic rel…
DecoderGeneralized Zero-Shot LearningGZSL Video ClassificationRetrieval+3