Papers AudioCaps
“AudioCaps” 태그가 달린 논문 64편 · 필터 해제
AC/DC: LLM-based Audio Comprehension via Dialogue Continuation
We propose an instruction-following audio comprehension model that leverages the dialogue continuation ability of large language models (LLMs). Instead of directly generating target captions in training data, the propose…
AudioCapsAudio captioningInstruction FollowingQuestion AnsweringGLAP: General contrastive audio-text pretraining across domains and languages
Contrastive Language Audio Pretraining (CLAP) is a widely-used method to bridge the gap between audio and text domains. Current CLAP methods enable sound and music retrieval in English, ignoring multilingual spoken conte…
AudioCapsKeyword SpottingRetrievalText RetrievalIMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
Text-to-audio generation synthesizes realistic sounds or music given a natural language prompt. Diffusion-based frameworks, including the Tango and the AudioLDM series, represent the state-of-the-art in text-to-audio gen…
AudioCapsAudio GenerationFADMitigating Audiovisual Mismatch in Visual-Guide Audio Captioning
Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-…
AudioCapsAudio captioningUncertainty QuantificationAudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
Diffusion models have significantly improved the quality and diversity of audio generation but are hindered by slow inference speed. Rectified flow enhances inference speed by learning straight-line ordinary differential…
AudioCapsAudio GenerationDiversityDiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining), have significantly enhanced the alignme…
AudioCapsAudio GenerationDenoisingQuality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model
Integrating audio and visual data for training multimodal foundational models remains challenging. We present Audio-Video Vector Alignment (AVVA), which aligns audiovisual (AV) scene content beyond mere temporal synchron…
AudioCapsContrastive LearningLarge Language ModelRetrieval+1TAIL: Text-Audio Incremental Learning
Many studies combine text and audio to capture multi-modal information but they overlook the model's generalization ability on new datasets. Introducing new datasets may affect the feature space of the original dataset, …
AudioCapsIncremental Learningtext similarityATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors
Multilingual audio-text retrieval (ML-ATR) is a challenging task that aims to retrieve audio clips or multilingual texts from databases. However, existing ML-ATR schemes suffer from inconsistencies for instance similarit…
AudioCapsContrastive LearningRetrievalText RetrievalUnbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning
Teacher-forcing training for audio captioning usually leads to exposure bias due to training and inference mismatch. Prior works propose the contrastive method to deal with caption degeneration. However, the contrastive …
AudioCapsAudio captioningADIFF: Explaining audio difference using natural language
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scen…
AudioCapsAudio captioningAudio GenerationLanguage Modeling+2LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
Automated audio captioning is a task that generates textual descriptions for audio content, and recent studies have explored using visual information to enhance captioning quality. However, current methods often fail to …
AudioCapsAudio captioningAudio-Visual CaptioningImage Captioning+3Language-based Audio Retrieval with Co-Attention Networks
In recent years, user-generated audio content has proliferated across various media platforms, creating a growing need for efficient retrieval methods that allow users to search for audio clips using natural language que…
AudioCapsLearning Semantic RepresentationsNatural Language QueriesRetrievalETTA: Elucidating the Design Space of Text-to-Audio Models
Recent years have seen significant progress in Text-To-Audio (TTA) synthesis, enabling users to enrich their creative workflows with synthetic audio generated from natural language prompts. Despite this progress, the eff…
AudioCapsAudio captioningAudio GenerationLanguage Modelling+2Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning
Retrieval-augmented generation can improve audio captioning by incorporating relevant audio-text pairs from a knowledge base. Existing methods typically rely solely on the input audio as a unimodal retrieval query. In co…
AudioCapsAudio captioningRAGRetrieval+1SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio unders…
AudioCapsAudio captioningCaption GenerationMachine Translation+4DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the …
AudioCapsRetrievalText RetrievalEnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance
In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with …
AudioCapsAudio captioningLanguage ModellingRerankingDissecting Temporal Understanding in Text-to-Audio Retrieval
Recent advancements in machine learning have fueled research on multimodal tasks, such as for instance text-to-video and text-to-audio retrieval. These tasks require models to understand the semantic content of video and…
AudioCapsRetrievalText to Audio RetrievalEstimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval
Dual-encoder-based audio retrieval systems are commonly optimized with contrastive learning on a set of matching and mismatching audio-caption pairs. This leads to a shared embedding space in which corresponding items fr…
AudioCapsContrastive LearningRetrievalText to Audio Retrieval