Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations
Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learning either global representations useful for high-level understanding tasks such as classification, or local representations useful for tasks such as audio-visual source localization and separation. While they produce satisfactory results in their intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose a versatile self-supervised approach to learn audio-visual representations that can generalize to both the tasks which require global semantic information (e.g. classification) and the tasks that require fine-grained spatio-temporal information (e.g. localization). We achieve this by optimizing two cross-modal contrastive objectives that together encourage our model to learn discriminative global-local visual information given the corresponding audio information. To show that our approach learns generalizable video representations, we evaluate it on various downstream scenarios including action/sound classification, lip reading, deepfake detection, and sound source localization.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationDeepFake DetectionFace SwappingGeneral ClassificationLip ReadingSelf-Supervised LearningSound ClassificationSound Source LocalizationSimilar Papers 제목 키워드 기반
Exploring Representation Learning for Small-Footprint Keyword Spotting
In this paper, we investigate representation learning for low-resource keyword spotting (KWS). The main challenges of KWS are limited labeled data and limited available device resources. To address those challenges, we e…
Contrastive LearningKeyword SpottingRepresentation LearningSmall-Footprint Keyword SpottingCAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
Recent advances in audio-visual learning have shown promising results in learning representations across modalities. However, most approaches rely on global audio representations that fail to capture fine-grained tempora…
audio-visual learningcross-modal alignmentMulti-network Contrastive Learning Based on Global and Local Representations
The popularity of self-supervised learning has made it possible to train models without relying on labeled data, which saves expensive annotation costs. However, most existing self-supervised contrastive learning methods…
Contrastive LearningLinear evaluationSelf-Supervised LearningContrastive Learning of Global and Local Video Representations
Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., global rep…
ClassificationContrastive LearningDeepFake DetectionFace Swapping+2Contrastive Learning of Global-Local Video Representations
Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., \textit{gl…
ClassificationContrastive LearningDeepFake DetectionFace Swapping+4