Efficient Audio Captioning Transformer with Patchout and Text Guidance

2023-04-06 · Thodoris Kouzelis, Grigoris Bastas, Athanasios Katsamanis, Alexandros Potamianos

Automated audio captioning is multi-modal translation task that aim to generate textual descriptions for a given audio clip. In this paper we propose a full Transformer architecture that utilizes Patchout as proposed in [1], significantly reducing the computational complexity and avoiding overfitting. The caption generation is partly conditioned on textual AudioSet tags extracted by a pre-trained classification model which is fine-tuned to maximize the semantic similarity between AudioSet labels and ground truth captions. To mitigate the data scarcity problem of Automated Audio Captioning we introduce transfer learning from an upstream audio-related task and an enlarged in-domain dataset. Moreover, we propose a method to apply Mixup augmentation for AAC. Ablation studies are carried out to investigate how Patchout and text guidance contribute to the final performance. The results show that the proposed techniques improve the performance of our system and while reducing the computational complexity. Our proposed method received the Judges Award at the Task6A of DCASE Challenge 2022.

📄 PDF Abstract BibTeX arXiv:2304.02916

Code (0)

등록된 구현이 없습니다.

Tasks

Audio captioningCaption GenerationSemantic Similarity Semantic Textual Similarity Transfer Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers

2022-08-24 · Paul Primus, Gerhard Widmer

Standard machine learning models for tagging and classifying acoustic signals cannot handle classes that were not seen during training. Zero-Shot (ZS) learning overcomes this restriction by predicting classes based on ad…

Audio Tagging Classification Environmental Sound Classification Sound Classification

Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation

2023-09-06 · Arvind Krishna Sridhar, Yinyi Guo, Erik Visser, Rehana Mahfuz

There has been significant research on developing pretrained transformer architectures for multimodal-to-text generation tasks. Albeit performance improvements, such models are frequently overparameterized, hence suffer …

Audio captioning Data Augmentation Hallucination Semantic Similarity+2

Zero-Shot Audio Captioning via Audibility Guidance

2023-09-07 · Tal Shaharabany, Ariel Shaulov, Lior Wolf

The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio -- (i) fluency of the generat…

Language Modeling Zero-shot Audio Captioning

Efficient Training of Audio Transformers with Patchout

2021-10-11 · Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, Gerhard Widmer

The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transfo…

Acoustic Scene Classification Audio Classification Audio Tagging GPU+1

Zero-shot audio captioning with audio-language model guidance and audio context keywords

2023-11-14 · Leonard Salewski, Stefan Fauth, A. Sophia Koepke, Zeynep Akata

Zero-shot audio captioning aims at automatically generating descriptive textual captions for audio content without prior training for this task. Different from speech recognition which translates audio content that conta…

Audio captioning Descriptive Image Captioning Language Modeling+5