DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates,query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.
Code (0)
등록된 구현이 없습니다.
Tasks
AudioCapsRetrievalText RetrievalSimilar Papers 제목 키워드 기반
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high…
Audio GenerationText-to-Video GenerationVideo GenerationTowards Diverse and Efficient Audio Captioning via Diffusion Models
We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achiev…
Audio captioningDiversityText GenerationImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model
We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained …
DescriptivemodelDreamAudio: Customized Text-to-Audio Generation with Diffusion Models
With the development of large-scale diffusion-based and language-modeling-based generative models, impressive progress has been achieved in text-to-audio generation. Despite producing high-quality outputs, existing text-…
Audio GenerationCRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis
In this paper, we propose a novel score-base generative model for unconditional raw audio synthesis. Our proposal builds upon the latest developments on diffusion process modeling with stochastic differential equations, …
Audio GenerationAudio SynthesisImage Generation