Weakly-Supervised Audio-Visual Segmentation
Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningSegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Weakly-supervised Audio-visual Sound Source Detection and Separation
Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from artificially mixed spectrograms, known as Mi…
Audio Source SeparationDenoisingObjectSegmentation+3From Waveforms to Pixels: A Survey on Audio-Visual Segmentation
Audio-Visual Segmentation (AVS) aims to identify and segment sound-producing objects in videos by leveraging both visual and audio modalities. It has emerged as a significant research area in multimodal perception, enabl…
Few-Shot LearningExploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
The audio-visual video parsing task aims to temporally parse a video into audio or visual event categories. However, it is labor intensive to temporally annotate audio and visual events and thus hampers the learning of a…
Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection
Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early …
Anomaly Detection In Surveillance Videosaudio-visual learningMultiple Instance LearningCross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization
Temporally localizing actions in videos is one of the key components for video understanding. Learning from weakly-labelled data is seen a potential solution towards avoiding expensive frame-level annotations. Different …
Action LocalizationVideo UnderstandingWeakly Supervised Action Localization