Hyperbolic Audio-visual Zero-shot Learning
Audio-visual zero-shot learning aims to classify samples consisting of a pair of corresponding audio and video sequences from classes that are not present during training. An analysis of the audio-visual data reveals a large degree of hyperbolicity, indicating the potential benefit of using a hyperbolic transformation to achieve curvature-aware geometric learning, with the aim of exploring more complex hierarchical data structures for this task. The proposed approach employs a novel loss function that incorporates cross-modality alignment between video and audio features in the hyperbolic space. Additionally, we explore the use of multiple adaptive curvatures for hyperbolic projections. The experimental results on this very challenging task demonstrate that our proposed hyperbolic approach for zero-shot learning outperforms the SOTA method on three datasets: VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL achieving a harmonic mean (HM) improvement of around 3.0%, 7.0%, and 5.3%, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
GZSL Video ClassificationZero-Shot LearningSimilar Papers 제목 키워드 기반
Hyperbolic Visual Embedding Learning for Zero-Shot Recognition
This paper proposes a Hyperbolic Visual Embedding Learning Network for zero-shot recognition. The network learns image embeddings in hyperbolic space, which is capable of preserving the hierarchical structure of semantic…
Zero-Shot LearningHAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA
Recent advances in representation learning have shown that hyperbolic geometry can offer a more expressive alternative to the Euclidean embeddings used in CLIP models, capturing hierarchical structures and leading to bet…
Visual Question AnsweringRepresentation LearningAudio-visual Generalized Zero-shot Learning the Easy Way
Audio-visual generalized zero-shot learning is a rapidly advancing domain that seeks to understand the intricate relations between audio and visual cues within videos. The overarching goal is to leverage insights from se…
Generalized Zero-Shot LearningZero-Shot LearningTemporal and cross-modal attention for audio-visual zero-shot learning
Audio-visual generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information in order to be able to recognise samples from novel, previously unseen cl…
GZSL Video ClassificationVideo ClassificationZero-Shot LearningZero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising
In this paper, we introduce zero-shot audio-video editing, a novel task that requires transforming original audio-visual content to align with a specified textual prompt without additional model training. To evaluate thi…
DenoisingVideo Editing