An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning
Automated audio captioning aims to use natural language to describe the content of audio data. This paper presents an audio captioning system with an encoder-decoder architecture, where the decoder predicts words based on audio features extracted by the encoder. To improve the proposed system, transfer learning from either an upstream audio-related task or a large in-domain dataset is introduced to mitigate the problem induced by data scarcity. Besides, evaluation metrics are incorporated into the optimization of the model with reinforcement learning, which helps address the problem of `exposure bias'' induced by `teacher forcing'' training strategy and the mismatch between the evaluation metrics and the loss function. The resulting system was ranked 3rd in DCASE 2021 Task 6. Ablation studies are carried out to investigate how much each element in the proposed system can contribute to final performance. The results show that the proposed techniques significantly improve the scores of the evaluation metrics, however, reinforcement learning may impact adversely on the quality of the generated captions.
Code (1)
Tasks
Audio captioningDecoderreinforcement-learningReinforcement LearningReinforcement Learning (RL)Transfer LearningSimilar Papers 제목 키워드 기반
Leveraging Pre-trained BERT for Audio Captioning
Audio captioning aims at using natural language to describe the content of an audio clip. Existing audio captioning systems are generally based on an encoder-decoder architecture, in which acoustic information is extract…
AudioCapsAudio captioningDecoderLanguage ModellingTraining Audio Captioning Models without Audio
Automated Audio Captioning (AAC) is the task of generating natural language descriptions given an audio stream. A typical AAC system requires manually curated training data of audio segments and corresponding text captio…
Audio captioningDecoderAudio Captioning Transformer
Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features…
AudioCapsAudio captioningDecoderRetrieval-augmented Few-shot In-context Audio CaptioningAudio Captioning with Composition of Acoustic and Semantic Information
Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the enc…
AudioCapsAudio captioningDecoderAutomated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization
In this paper, we examine the use of Transfer Learning using Pretrained Audio Neural Networks (PANNs), and propose an architecture that is able to better leverage the acoustic features provided by PANNs for the Automated…
Audio captioningDecoderTransfer Learning