DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining), have significantly enhanced the alignment of text, video, and audio embeddings via a single contrastive loss. However, these methods often overlook the bidirectional interactions and inherent noises present in each modality, which can crucially impact the quality and efficacy of cross-modal integration. To address this limitation, we introduce DiffGAP, a novel approach incorporating a lightweight generative module within the contrastive space. Specifically, our DiffGAP employs a bidirectional diffusion process tailored to bridge the cross-modal gap more effectively. This involves a denoising process on text and video embeddings conditioned on audio embeddings and vice versa, thus facilitating a more nuanced and robust cross-modal interaction. Our experimental results on VGGSound and AudioCaps datasets demonstrate that DiffGAP significantly improves performance in video/text-audio generation and retrieval tasks, confirming its effectiveness in enhancing cross-modal understanding and generation capabilities.
Code (0)
등록된 구현이 없습니다.
Tasks
AudioCapsAudio GenerationDenoisingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GIDN: A Lightweight Graph Inception Diffusion Network for High-efficient Link Prediction
In this paper, we propose a Graph Inception Diffusion Networks(GIDN) model. This model generalizes graph diffusion in different feature spaces, and uses the inception module to avoid the large amount of computations caus…
Link PredictionDR-BFR: Degradation Representation with Diffusion Models for Blind Face Restoration
Blind face restoration (BFR) is fundamentally challenged by the extensive range of degradation types and degrees that impact model generalization. Recent advancements in diffusion models have made considerable progress i…
Blind Face RestorationContrastive LearningDenoisingPainterly Image Harmonization using Diffusion Model
Painterly image harmonization aims to insert photographic objects into paintings and obtain artistically coherent composite images. Previous methods for this task mainly rely on inference optimization or generative adver…
Generative Adversarial NetworkImage HarmonizationInference OptimizationmodelProbing Diffusion Denoising Dynamics for Contrastive Representation Learning
Text-to-image diffusion models exhibit unprecedented generative capability and contain rich intermediate representations that can be useful for discriminative vision tasks. Motivated by this observation, we study a focus…
Representation LearningContrastive LearningDiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
Text-to-audio (TTA) generation is a recent popular problem that aims to synthesize general audio given text descriptions. Previous methods utilized latent diffusion models to learn audio embedding in a latent space with …
AudioCapsAudio GenerationContrastive Learning