Weakly-Supervised Action Detection Guided by Audio Narration
Videos are more well-organized curated data sources for visual concept learning than images. Unlike the 2-dimensional images which only involve the spatial information, the additional temporal dimension bridges and synchronizes multiple modalities. However, in most video detection benchmarks, these additional modalities are not fully utilized. For example, EPIC Kitchens is the largest dataset in first-person (egocentric) vision, yet it still relies on crowdsourced information to refine the action boundaries to provide instance-level action annotations. We explored how to eliminate the expensive annotations in video detection data which provide refined boundaries. We propose a model to learn from the narration supervision and utilize multimodal features, including RGB, motion flow, and ambient sound. Our model learns to attend to the frames related to the narration label while suppressing the irrelevant frames from being used. Our experiments show that noisy audio narration suffices to learn a good action detection model, thus reducing annotation expenses.
Code (0)
등록된 구현이 없습니다.
Tasks
Action DetectionSimilar Papers 제목 키워드 기반
Guided learning for weakly-labeled semi-supervised sound event detection
We propose a simple but efficient method termed Guided Learning for weakly-labeled semi-supervised sound event detection (SED). There are two sub-targets implied in weakly-labeled SED: audio tagging and boundary detectio…
Audio TaggingBoundary DetectionEvent DetectionGeneral Classification+1Weakly-supervised Audio-visual Sound Source Detection and Separation
Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from artificially mixed spectrograms, known as Mi…
Audio Source SeparationDenoisingObjectSegmentation+3Past and Future Motion Guided Network for Audio Visual Event Localization
In recent years, audio-visual event localization has attracted much attention. It's purpose is to detect the segment containing audio-visual events and recognize the event category from untrimmed videos. Existing methods…
audio-visual event localizationAudio-Guided Attention Network for Weakly Supervised Violence Detection
Detecting violence in video is a challenging task due to its complex scenarios and great intra-class variability. Most previous works specialize in the analysis of appearance or motion information, ignoring the co-occurr…
Anomaly Detection In Surveillance VideosWeakly Supervised Scalable Audio Content Analysis
Audio Event Detection is an important task for content analysis of multimedia data. Most of the current works on detection of audio events is driven through supervised learning approaches. We propose a weakly supervised …
Event DetectionMultiple Instance LearningWeakly-supervised Learning