Audio Difference Learning for Audio Captioning
This study introduces a novel training paradigm, audio difference learning, for improving audio captioning. The fundamental concept of the proposed learning method is to create a feature representation space that preserves the relationship between audio, enabling the generation of captions that detail intricate audio information. This method employs a reference audio along with the input audio, both of which are transformed into feature representations via a shared encoder. Captions are then generated from these differential features to describe their differences. Furthermore, a unique technique is proposed that involves mixing the input audio with additional audio, and using the additional audio as a reference. This results in the difference between the mixed audio and the reference audio reverting back to the original input audio. This allows the original input's caption to be used as the caption for their difference, eliminating the need for additional annotations for the differences. In the experiments using the Clotho and ESC50 datasets, the proposed method demonstrated an improvement in the SPIDEr score by 7% compared to conventional methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio captioningSimilar Papers 제목 키워드 기반
Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement
We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the prob…
Audio captioningDisentanglementADIFF: Explaining audio difference using natural language
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scen…
AudioCapsAudio captioningAudio GenerationLanguage Modeling+2Temporal Sub-sampling of Audio Feature Sequences for Automated Audio Captioning
Audio captioning is the task of automatically creating a textual description for the contents of a general audio signal. Typical audio captioning methods rely on deep neural networks (DNNs), where the target of the DNN i…
Audio captioningAn investigation on selecting audio pre-trained models for audio captioning
Audio captioning is a task that generates description of audio based on content. Pre-trained models are widely used in audio captioning due to high complexity. Unless a comprehensive system is re-trained, it is hard to d…
Audio captioningInteractive Audio-text Representation for Automated Audio Captioning with Contrastive Learning
Automated Audio captioning (AAC) is a cross-modal task that generates natural language to describe the content of input audio. Most prior works usually extract single-modality acoustic features and are therefore sub-opti…
Audio captioningContrastive Learning