Self-supervised Contrastive Learning for Audio-Visual Action Recognition

2022-04-28 · Yang Liu, Ying Tan, Haoyuan Lan

The underlying correlation between audio and visual modalities can be utilized to learn supervised information for unlabeled videos. In this paper, we propose an end-to-end self-supervised framework named Audio-Visual Contrastive Learning (AVCL), to learn discriminative audio-visual representations for action recognition. Specifically, we design an attention based multi-modal fusion module (AMFM) to fuse audio and visual modalities. To align heterogeneous audio-visual modalities, we construct a novel co-correlation guided representation alignment module (CGRA). To learn supervised information from unlabeled videos, we propose a novel self-supervised contrastive learning module (SelfCL). Furthermore, we build a new audio-visual action recognition dataset named Kinetics-Sounds100. Experimental results on Kinetics-Sounds32 and Kinetics-Sounds100 datasets demonstrate the superiority of our AVCL over the state-of-the-art methods on large-scale action recognition benchmark.

📄 PDF Abstract BibTeX arXiv:2204.13386

Code (0)

등록된 구현이 없습니다.

Tasks

Action Recognition Contrastive Learning Self-Supervised Action Recognition

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Contrastive Learning 설명 없음

Contrastive Audio-Visual Masked Autoencoder

2022-10-02 · Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath 외

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by co…

Audio Classification Audio Tagging Contrastive Learning Multi-modal Classification+4

Contrastive Self-Supervised Learning of Global-Local Audio-Visual Representations

2021-01-01 · Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song

Contrastive self-supervised learning has delivered impressive results in many audio-visual recognition tasks. However, existing approaches optimize for learning either global representations useful for high-level underst…

Classification DeepFake Detection Face Swapping General Classification+4

Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection

2022-07-12 · Jiashuo Yu, Jinyu Liu, Ying Cheng, Rui Feng 외

Weakly-supervised audio-visual violence detection aims to distinguish snippets containing multimodal violence events with video-level labels. Many prior works perform audio-visual integration and interaction in an early …

Anomaly Detection In Surveillance Videos audio-visual learning Multiple Instance Learning

HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition

2024-01-11 · Licai Sun, Zheng Lian, Bin Liu, JianHua Tao

Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised…

Contrastive Learning Dynamic Facial Expression Recognition Emotion Recognition Representation Learning+1

SyncLipMAE: Contrastive Masked Pretraining for Audio-Visual Talking-Face Representation

2025-10-11 · Zeyu Ling, Xiaodong Gu, Jiangnan Tang, Changqing Zou arxiv

We introduce SyncLipMAE, a self-supervised pretraining framework for talking-face video that learns synchronization-aware and transferable facial dynamics from unlabeled audio-visual streams. Our approach couples masked …

Visual Speech Recognition Action Recognition