Heterogeneous Graph Learning for Acoustic Event Classification
Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous graphs an attractive option. However, graph structure does not appear naturally in audiovisual data. Graphs for audiovisual data are constructed manually which is both difficult and sub-optimal. In this work, we address this problem by (i) proposing a parametric graph construction strategy for the intra-modal edges, and (ii) learning the crossmodal edges. To this end, we develop a new model, heterogeneous graph crossmodal network (HGCN) that learns the crossmodal edges. Our proposed model can adapt to various spatial and temporal scales owing to its parametric construction, while the learnable crossmodal edges effectively connect the relevant nodes across modalities. Experiments on a large benchmark dataset (AudioSet) show that our model is state-of-the-art (0.53 mean average precision), outperforming transformer-based models and other graph-based models.
Code (1)
Tasks
Classificationgraph constructionGraph LearningSimilar Papers 제목 키워드 기반
Visually-aware Acoustic Event Detection using Heterogeneous Graphs
Perception of auditory events is inherently multimodal relying on both audio and visual cues. A large number of existing multimodal approaches process each modality using modality-specific models and then fuse the embedd…
Event DetectionMulti-dimensional Edge-based Audio Event Relational Graph Representation Learning for Acoustic Scene Classification
Most existing deep learning-based acoustic scene classification (ASC) approaches directly utilize representations extracted from spectrograms to identify target scenes. However, these approaches pay little attention to t…
Acoustic Scene ClassificationGraph Representation LearningRepresentation LearningScene ClassificationAudio Event-Relational Graph Representation Learning for Acoustic Scene Classification
Most deep learning-based acoustic scene classification (ASC) approaches identify scenes based on acoustic features converted from audio clips containing mixed information entangled by polyphonic audio events (AEs). Howev…
Acoustic Scene ClassificationGraph Representation LearningRepresentation LearningScene ClassificationCNN-based Discriminative Training for Domain Compensation in Acoustic Event Detection with Frame-wise Classifier
Domain mismatch is a noteworthy issue in acoustic event detection tasks, as the target domain data is difficult to access in most real applications. In this study, we propose a novel CNN-based discriminative training fra…
Event DetectionTMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification
Audiovisual data is everywhere in this digital age, which raises higher requirements for the deep learning models developed on them. To well handle the information of the multi-modal data is the key to a better audiovisu…
Graph Learning