Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition
Humans do not understand individual events in isolation; rather, they generalize concepts within classes and compare them to others. Existing audio-video pre-training paradigms only focus on the alignment of the overall audio-video modalities, without considering the reinforcement of distinguishing easily confused classes through cognitive induction and contrast during training. This paper proposes the Dynamic Inter-Class Confusion-Aware Encoder (DICCAE), an encoder that aligns audio-video representations at a fine-grained, category-level. DICCAE addresses category confusion by dynamically adjusting the confusion loss based on inter-class confusion degrees, thereby enhancing the model's ability to distinguish between similar activities. To further extend the application of DICCAE, we also introduce a novel training framework that incorporates both audio and video modalities, as well as their fusion. To mitigate the scarcity of audio-video data in the human activity recognition task, we propose a cluster-guided audio-video self-supervised pre-training strategy for DICCAE. DICCAE achieves near state-of-the-art performance on the VGGSound dataset, with a top-1 accuracy of 65.5%. We further evaluate its feature representation quality through extensive ablation studies, validating the necessity of each module.
Code (0)
등록된 구현이 없습니다.
Tasks
Human Activity RecognitionSimilar Papers 제목 키워드 기반
Reducing Class-wise Confusion for Incremental Learning with Disentangled Manifolds
Class incremental learning (CIL) aims to enable models to continuously learn new classes without catastrophically forgetting old ones. A promising direction is to learn and use prototypes of classes during incremental up…
class-incremental learningClass Incremental LearningIncremental LearningResolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning
The dynamic expansion architecture is becoming popular in class incremental learning, mainly due to its advantages in alleviating catastrophic forgetting. However, task confusion is not well assessed within this framewor…
class-incremental learningClass Incremental LearningIncremental LearningIncremental Learning on CIFAR100-B0(5steps of 20 classes)+1Closing the Confusion Loop: CLIP-Guided Alignment for Source-Free Domain Adaptation
Source-Free Domain Adaptation (SFDA) tackles the problem of adapting a pre-trained source model to an unlabeled target domain without accessing any source data, which is quite suitable for the field of data security. Alt…
Source-Free Domain AdaptationContrastive LearningCAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
Vision-language models like CLIP have achieved remarkable progress in cross-modal representation learning, yet suffer from systematic misclassifications among visually and semantically similar categories. We observe that…
Representation LearningLanguage-Aware Token Boosting: LLM Language Confusion Reduction Without Tuning
Large language models (LLMs) sometimes exhibit language confusion when generating non-English text. Existing approaches typically rely on fine-tuning to mitigate this issue. In contrast, we propose a tuning-free paradigm…