A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification
In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to effectively combine the video and audio modalities. Moreover, data augmentation strategies are investigated to increase audio-visual training set size. For the video modality the effectiveness of several operations in RandAugment is verified. An audio-video joint mixup scheme is proposed to further improve AVSC performances. Evaluated on the development set of TAU Urban Audio Visual Scenes 2021, our final system can achieve the best accuracy of 94.2% among all single AVSC systems submitted to DCASE 2021 Task 1b.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationScene ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
A class of recent semi-supervised learning (SSL) methods heavily rely on domain-specific data augmentations. In contrast, generative SSL methods involve unsupervised learning based on generative models by either joint-tr…
image-classificationImage ClassificationDifferentiable Retrieval Augmentation via Generative Language Modeling for E-commerce Query Intent Classification
Retrieval augmentation, which enhances downstream models by a knowledge retriever and an external corpus instead of by merely increasing the number of model parameters, has been successfully applied to many natural langu…
intent-classificationIntent ClassificationLanguage ModelingLanguage Modelling+4Tradeoffs in Data Augmentation: An Empirical Study
Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice, augmentation policie…
Data AugmentationDiversityExploring the Efficacy of Base Data Augmentation Methods in Deep Learning-Based Radiograph Classification of Knee Joint Osteoarthritis
Diagnosing knee joint osteoarthritis (KOA), a major cause of disability worldwide, is challenging due to subtle radiographic indicators and the varied progression of the disease. Using deep learning for KOA diagnosis req…
Data AugmentationTHeGAU: Type-Aware Heterogeneous Graph Autoencoder and Augmentation
Heterogeneous Graph Neural Networks (HGNNs) are effective for modeling Heterogeneous Information Networks (HINs), which encode complex multi-typed entities and relations. However, HGNNs often suffer from type information…
Node Classification