Zero-shot Audio Captioning
2개 벤치마크 · 논문 8편 · 이 태스크의 논문 보기 →
Benchmarks
Most implemented
DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
Papers
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
Contrastive Language-Audio Pretraining (CLAP) models are widely used for audio understanding and support modality-agnostic condition swapping in many zero-shot applications. However, their performance is heavily affected…
Zero-shot Audio CaptioningDimensionality ReductionMAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models
Automated Audio Captioning (AAC) generates captions for audio clips but faces challenges due to limited datasets compared to image captioning. To overcome this, we propose the zero-shot AAC system that leverages pre-trai…
Zero-shot Audio CaptioningImage CaptioningClassifier-Guided Captioning Across Modalities
Most current captioning systems use language models trained on data from specific settings, such as image-based captioning via Amazon Mechanical Turk, limiting their ability to generalize to other modality distributions …
Audio captioningVideo CaptioningZero-shot Audio CaptioningDRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning
While automated audio captioning (AAC) has made notable progress, traditional fully supervised AAC models still face two critical challenges: the need for expensive audio-text pair data for training and performance degra…
Audio captioningLarge Language ModelRetrievalRetrieval-augmented Generation+1An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts. In this work, we show that this ability can b…
Audio captioningContrastive LearningImage CaptioningRepresentation Learning+1Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Fla…
Acoustic Scene ClassificationAudio captioningFew-Shot LearningIn-Context Learning+5