Audio-Visual Class-Incremental Learning

2023-08-21 · ICCV 2023 1 · Weiguo Pian, Shentong Mo, Yunhui Guo, Yapeng Tian

In this paper, we introduce audio-visual class-incremental learning, a class-incremental learning scenario for audio-visual video recognition. We demonstrate that joint audio-visual modeling can improve class-incremental learning, but current methods fail to preserve semantic similarity between audio and visual features as incremental step grows. Furthermore, we observe that audio-visual correlations learned in previous tasks can be forgotten as incremental steps progress, leading to poor performance. To overcome these challenges, we propose AV-CIL, which incorporates Dual-Audio-Visual Similarity Constraint (D-AVSC) to maintain both instance-aware and class-aware semantic similarity between audio-visual modalities and Visual Attention Distillation (VAD) to retain previously learned audio-guided visual attentive ability. We create three audio-visual class-incremental datasets, AVE-Class-Incremental (AVE-CI), Kinetics-Sounds-Class-Incremental (K-S-CI), and VGGSound100-Class-Incremental (VS100-CI) based on the AVE, Kinetics-Sounds, and VGGSound datasets, respectively. Our experiments on AVE-CI, K-S-CI, and VS100-CI demonstrate that AV-CIL significantly outperforms existing class-incremental learning methods in audio-visual class-incremental learning. Code and data are available at: https://github.com/weiguoPian/AV-CIL_ICCV2023.

📄 PDF Abstract BibTeX arXiv:2308.11073

Code (1)

weiguopian/av-cil_iccv2023 공식 구현 pytorch

Tasks

class-incremental learning Class Incremental Learning Incremental Learning Semantic Similarity Semantic Textual SimilarityVideo Recognition

Methods 이 논문이 사용한 방법론

fail 설명 없음

Class-Incremental Grouping Network for Continual Audio-Visual Learning

2023-09-11 · ICCV 2023 1 · Shentong Mo, Weiguo Pian, Yapeng Tian

Continual learning is a challenging problem in which models need to be trained on non-stationary data across sequential tasks for class-incremental learning. While previous methods have focused on using either regulariza…

audio-visual learning class-incremental learning Class Incremental Learning Continual Learning+3

Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio

2026-06-09 · Avi Gupta, Nilotpal Sinha, Vishnu Raj, Sambuddha Saha 외 arxiv

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. While recent CIL advances have spurred significant interest across various modalities, the audio-v…

class-incremental learning

Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture

2025-04-21 · Meng Cui, Xianghu Yue, Xinyuan Qian, Jinzheng Zhao 외

Fish Feeding Intensity Assessment (FFIA) is crucial in industrial aquaculture management. Recent multi-modal approaches have shown promise in improving FFIA robustness and efficiency. However, these methods face signific…

Benchmarking class-incremental learning Class Incremental Learning Exemplar-Free+1

Leveraging Pre-Trained Models for Multimodal Class-Incremental Learning under Adaptive Fusion

2025-02-07 · Yukun Chen, Zihuan Qiu, Fanman Meng, Hongliang Li 외

Unlike traditional Multimodal Class-Incremental Learning (MCIL) methods that focus only on vision and text, this paper explores MCIL across vision, audio and text modalities, addressing challenges in integrating compleme…

class-incremental learning Class Incremental Learning cross-modal alignment Diversity+2

Domain-Incremental Learning for Audio Classification

2024-12-23 · Manjunath Mulimani, Annamaria Mesaros

In this work, we propose a method for domain-incremental learning for audio classification from a sequence of datasets recorded in different acoustic conditions. Fine-tuning a model on a sequence of evolving domains or d…

Audio Classification Classification Incremental Learning Multi-Label Classification+1