CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling
We introduce a multi-modal diffusion model tailored for the bi-directional conditional generation of video and audio. We propose a joint contrastive training loss to improve the synchronization between visual and auditory occurrences. We present experiments on two datasets to evaluate the efficacy of our proposed model. The assessment of generation quality and alignment performance is carried out from various angles, encompassing both objective and subjective metrics. Our findings demonstrate that the proposed model outperforms the baseline in terms of quality and generation speed through introduction of our novel cross-modal easy fusion architectural block. Furthermore, the incorporation of the contrastive loss results in improvements in audio-visual alignment, particularly in the high-correlation video-to-audio generation task.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Bridging Contrastive Learning and Domain Adaptation: Theoretical Perspective and Practical Application
This work studies the relationship between Contrastive Learning and Domain Adaptation from a theoretical perspective. The two standard contrastive losses, NT-Xent loss (Self-supervised) and Supervised Contrastive loss, a…
Contrastive LearningDomain AdaptationGMS-CAVP: Improving Audio-Video Correspondence with Multi-Scale Contrastive and Generative Pretraining
Recent advances in video-audio (V-A) understanding and generation have increasingly relied on joint V-A embeddings, which serve as the foundation for tasks such as cross-modal retrieval and generation. While prior method…
Cross-Modal RetrievalContrastive LearningDiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining), have significantly enhanced the alignme…
AudioCapsAudio GenerationDenoisingMultimodal Generative Models for Bankruptcy Prediction Using Textual Data
Textual data from financial filings, e.g., the Management's Discussion & Analysis (MDA) section in Form 10-K, has been used to improve the prediction accuracy of bankruptcy models. In practice, however, we cannot obtain …
AllPredictionMeasuring Differences between Conditional Distributions using Kernel Embeddings
Comparing conditional distributions is a fundamental challenge in statistics and machine learning, with applications across a wide range of domains. While proposed methods for measuring discrepancies using kernel embeddi…