Audio Captioning Transformer
Audio captioning aims to automatically generate a natural language description of an audio clip. Most captioning models follow an encoder-decoder architecture, where the decoder predicts words based on the audio features extracted by the encoder. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are often used as the audio encoder. However, CNNs can be limited in modelling temporal relationships among the time frames in an audio signal, while RNNs can be limited in modelling the long-range dependencies among the time frames. In this paper, we propose an Audio Captioning Transformer (ACT), which is a full Transformer network based on an encoder-decoder architecture and is totally convolution-free. The proposed method has a better ability to model the global information within an audio signal as well as capture temporal relationships between audio events. We evaluate our model on AudioCaps, which is the largest audio captioning dataset publicly available. Our model shows competitive performance compared to other state-of-the-art approaches.
Code (1)
Tasks
AudioCapsAudio captioningDecoderRetrieval-augmented Few-shot In-context Audio CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement
We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the prob…
Audio captioningDisentanglementWaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information
Automated audio captioning (AAC) is a novel task, where a method takes as an input an audio sample and outputs a textual description (i.e. a caption) of its contents. Most AAC methods are adapted from from image captioni…
Audio captioningDecoderImage CaptioningMachine Translation+1A Whisper transformer for audio captioning trained with synthetic captions and transfer learning
The field of audio captioning has seen significant advancements in recent years, driven by the availability of large-scale audio datasets and advancements in deep learning techniques. In this technical report, we present…
Audio captioningSpeech-to-TextTransfer LearningA Transformer-based Audio Captioning Model with Keyword Estimation
One of the problems with automated audio captioning (AAC) is the indeterminacy in word selection corresponding to the audio event/scene. Since one acoustic event/scene can be described with several words, it results in a…
Acoustic Scene ClassificationAudio captioningEvent DetectionScene ClassificationDual Transformer Decoder based Features Fusion Network for Automated Audio Captioning
Automated audio captioning (AAC) which generates textual descriptions of audio content. Existing AAC models achieve good results but only use the high-dimensional representation of the encoder. There is always insufficie…
Audio captioningDecoder