Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation

2022-11-09 · Florian Schmid, Khaled Koutini, Gerhard Widmer

Audio Spectrogram Transformer models rule the field of Audio Tagging, outrunning previously dominating Convolutional Neural Networks (CNNs). Their superiority is based on the ability to scale up and exploit large-scale datasets such as AudioSet. However, Transformers are demanding in terms of model size and computational requirements compared to CNNs. We propose a training procedure for efficient CNNs based on offline Knowledge Distillation (KD) from high-performing yet complex transformers. The proposed training schema and the efficient CNN design based on MobileNetV3 results in models outperforming previous solutions in terms of parameter and computational efficiency and prediction performance. We provide models of different complexity levels, scaling from low-complexity models up to a new state-of-the-art performance of .483 mAP on AudioSet. Source Code available at: https://github.com/fschmid56/EfficientAT

📄 PDF Abstract BibTeX arXiv:2211.04772

Code (2)

fschmid56/efficientat 공식 구현 pytorch

fschmid56/efficientat_hear pytorch

Tasks

Audio Classification Audio Tagging Computational Efficiency Knowledge Distillation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

Batch Normalization 설명 없음

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Sigmoid Activation 설명 없음

Depthwise Separable Convolution While standard convolution performs the channelwise and spatial-wise computation in one step, Depthwise Separable Convolution …

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models

2023-10-24 · Florian Schmid, Khaled Koutini, Gerhard Widmer

The introduction of large-scale audio datasets, such as AudioSet, paved the way for Transformers to conquer the audio domain and replace CNNs as the state-of-the-art neural network architecture for many tasks. Audio Spec…

Audio Classification Audio Tagging Instrument Recognition Knowledge Distillation

Audio Mamba: Pretrained Audio State Space Model For Audio Tagging

2024-05-22 · Jiaju Lin, Haoxuan Hu

Audio tagging is an important task of mapping audio samples to their corresponding categories. Recently endeavours that exploit transformer models in this field have achieved great success. However, the quadratic self-at…

Audio Tagging Mamba State Space Models

DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners

2024-07-04 · Saurabhchand Bhati, Yuan Gong, Leonid Karlinsky, Hilde Kuehne 외

State-space models (SSMs) have emerged as an alternative to Transformers for audio modeling due to their high computational efficiency with long inputs. While recent efforts on Audio SSMs have reported encouraging result…

Audio Classification Audio Tagging Computational Efficiency Knowledge Distillation+1

Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers

2022-08-24 · Paul Primus, Gerhard Widmer

Standard machine learning models for tagging and classifying acoustic signals cannot handle classes that were not seen during training. Zero-Shot (ZS) learning overcomes this restriction by predicting classes based on ad…

Audio Tagging Classification Environmental Sound Classification Sound Classification

ATGNN: Audio Tagging Graph Neural Network

2023-11-02 · Shubhr Singh, Christian J. Steinmetz, Emmanouil Benetos, Huy Phan 외

Deep learning models such as CNNs and Transformers have achieved impressive performance for end-to-end audio tagging. Recent works have shown that despite stacking multiple layers, the receptive field of CNNs remains sev…

Audio Tagging Graph Neural Network