Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement
We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the problem that conventional audio captioning sometimes generates similar captions for similar audio clips, failing to describe the difference in content. We also propose a cross-attention-concentrated transformer encoder to extract differences by comparing a pair of audio clips and a similarity-discrepancy disentanglement to emphasize the difference in the latent space. To evaluate the proposed methods, we built an AudioDiffCaps dataset consisting of pairs of similar but slightly different audio clips with human-annotated descriptions of their differences. The experiment with the AudioDiffCaps dataset showed that the proposed methods solve the ADC task effectively and improve the attention weights to extract the difference by visualizing them in the transformer encoder.
Code (1)
Tasks
Audio captioningDisentanglementSimilar Papers 제목 키워드 기반
Audio Difference Learning for Audio Captioning
This study introduces a novel training paradigm, audio difference learning, for improving audio captioning. The fundamental concept of the proposed learning method is to create a feature representation space that preserv…
Audio captioningParameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation
There has been significant research on developing pretrained transformer architectures for multimodal-to-text generation tasks. Albeit performance improvements, such models are frequently overparameterized, hence suffer …
Audio captioningData AugmentationHallucinationSemantic Similarity+2ADIFF: Explaining audio difference using natural language
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scen…
AudioCapsAudio captioningAudio GenerationLanguage Modeling+2Semantic-Aware Confidence Calibration for Automated Audio Captioning
Automated audio captioning models frequently produce overconfident predictions regardless of semantic accuracy, limiting their reliability in deployment. This deficiency stems from two factors: evaluation metrics based o…
Semantic SimilarityAudio captioningAutomated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization
In this paper, we examine the use of Transfer Learning using Pretrained Audio Neural Networks (PANNs), and propose an architecture that is able to better leverage the acoustic features provided by PANNs for the Automated…
Audio captioningDecoderTransfer Learning