Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval
The goal of audio captioning is to translate input audio into its description using natural language. One of the problems in audio captioning is the lack of training data due to the difficulty in collecting audio-caption pairs by crawling the web. In this study, to overcome this problem, we propose to use a pre-trained large-scale language model. Since an audio input cannot be directly inputted into such a language model, we utilize guidance captions retrieved from a training dataset based on similarities that may exist in different audio. Then, the caption of the audio input is generated by using a pre-trained language model while referring to the guidance captions. Experimental results show that (i) the proposed method has succeeded to use a pre-trained language model for audio captioning, and (ii) the oracle performance of the pre-trained model-based caption generator was clearly better than that of the conventional method trained from scratch.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio captioningLanguage ModelingLanguage ModellingRetrievalSimilar Papers 제목 키워드 기반
Leveraging Pre-trained BERT for Audio Captioning
Audio captioning aims at using natural language to describe the content of an audio clip. Existing audio captioning systems are generally based on an encoder-decoder architecture, in which acoustic information is extract…
AudioCapsAudio captioningDecoderLanguage ModellingAudioCaps: Generating Captions for Audios in The Wild
We explore the problem of Audio Captioning: generating natural language description for any kind of audio in the wild, which has been surprisingly unexplored in previous research. We contribute a large-scale dataset of 4…
AudioCapsAudio captioningZero-shot audio captioning with audio-language model guidance and audio context keywords
Zero-shot audio captioning aims at automatically generating descriptive textual captions for audio content without prior training for this task. Different from speech recognition which translates audio content that conta…
Audio captioningDescriptiveImage CaptioningLanguage Modeling+5Prefix tuning for automated audio captioning
Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, w…
AudioCapsAudio captioningLanguage ModelingLanguage Modelling+2AUTOMATED AUDIO CAPTIONING BY FINE-TUNING BART WITH AUDIOSET TAGS
utomated audio captioning is the multimodal task of describing environmental audio recordings with fluent natural language. Most current methods utilize pre-trained analysis models to extract rele- vant semantic conte…
AudioCapsAudio captioningCaption GenerationLanguage Modeling+2