Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners

2023-06-01 · Sarthak Yadav, Sergios Theodoridis, Lars Kai Hansen, Zheng-Hua Tan

In this work, we propose a Multi-Window Masked Autoencoder (MW-MAE) fitted with a novel Multi-Window Multi-Head Attention (MW-MHA) module that facilitates the modelling of local-global interactions in every decoder transformer block through attention heads of several distinct local and global windows. Empirical results on ten downstream audio tasks show that MW-MAEs consistently outperform standard MAEs in overall performance and learn better general-purpose audio representations, along with demonstrating considerably better scaling characteristics. Investigating attention distances and entropies reveals that MW-MAE encoders learn heads with broader local and global attention. Analyzing attention head feature representations through Projection Weighted Canonical Correlation Analysis (PWCCA) shows that attention heads with the same window sizes across the decoder layers of the MW-MAE learn correlated feature representations which enables each block to independently capture local and global information, leading to a decoupled decoder feature hierarchy. Code for feature extraction and downstream experiments along with pre-trained models will be released publically.

📄 PDF Abstract BibTeX arXiv:2306.00561

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Masked Autoencoders that Listen

2022-07-13 · Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski 외

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE fi…

Audio Classification Decoder Representation Learning Speaker Identification

Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoders

2023-12-17 · Yaohua Zha, Huizhen Ji, Jinmin Li, Rongsheng Li 외

Learning 3D representation plays a critical role in masked autoencoder (MAE) based pre-training methods for point cloud, including single-modal and cross-modal based MAE. Specifically, although cross-modal MAE methods le…

3D Point Cloud Classification Few-Shot 3D Point Cloud Classification

DenoMAE2.0: Improving Denoising Masked Autoencoders by Classifying Local Patches

2025-02-25 · Atik Faysal, Mohammad Rostami, Taha Boushine, Reihaneh Gh. Roshan 외

We introduce DenoMAE2.0, an enhanced denoising masked autoencoder that integrates a local patch classification objective alongside traditional reconstruction loss to improve representation learning and robustness. Unlike…

Classification Denoising Representation Learning

SupMAE: Supervised Masked Autoencoders Are Efficient Vision Learners

2022-05-28 · Feng Liang, Yangguang Li, Diana Marculescu

Recently, self-supervised Masked Autoencoders (MAE) have attracted unprecedented attention for their impressive representation learning ability. However, the pretext task, Masked Image Modeling (MIM), reconstructs the mi…

Representation Learning Transfer Learning

Masked Autoencoders as Image Processors

2023-03-30 · Huiyu Duan, Wei Shen, Xiongkuo Min, Danyang Tu 외

Transformers have shown significant effectiveness for various vision tasks including both high-level vision and low-level vision. Recently, masked autoencoders (MAE) for feature pre-training have further unleashed the po…

Deblurring Denoising Image Defocus Deblurring Image Denoising+1