Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding mechanism of masked autoencoders (MAE). We decompose this decoding mechanism for masked patch reconstruction in MAE into self-attention and cross-attention. Our investigations suggest that self-attention between mask patches is not essential for learning good representations. To this end, we propose a novel pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE). CrossMAE's decoder leverages only cross-attention between masked and visible tokens, with no degradation in downstream performance. This design also enables decoding only a small subset of mask tokens, boosting efficiency. Furthermore, each decoder block can now leverage different encoder features, resulting in improved representation learning. CrossMAE matches MAE in performance with 2.5 to 3.7$\times$ less decoding compute. It also surpasses MAE on ImageNet classification and COCO instance segmentation under the same compute. Code and models: https://crossmae.github.io
Code (2)
Tasks
DecoderInstance SegmentationRepresentation LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Focus on Texture: Rethinking Pre-training in Masked Autoencoders for Medical Image Classification
Masked Autoencoders (MAEs) have emerged as a dominant strategy for self-supervised representation learning in natural images, where models are pre-trained to reconstruct masked patches with a pixel-wise mean squared erro…
Gallbladder Cancer DetectionMedical Image ClassificationUnsupervised Pre-trainingRepresentation LearningMasked Autoencoders in 3D Point Cloud Representation Learning
Transformer-based Self-supervised Representation Learning methods learn generic features from unlabeled datasets for providing useful network initialization parameters for downstream tasks. Recently, self-supervised lear…
Point Cloud CompletionPoint cloud reconstructionRepresentation LearningSelf-Supervised LearningMasked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input
Masked Autoencoders is a simple yet powerful self-supervised learning method. However, it learns representations indirectly by reconstructing masked input patches. Several methods learn representations directly by predic…
Audio ClassificationAudio TaggingKeyword SpottingKeyword Spotting on Google Speech Commands+3PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders
Masked autoencoder has been widely explored in point cloud self-supervised learning, whereby the point cloud is generally divided into visible and masked parts. These methods typically include an encoder accepting visibl…
3D Object Classification3D Point Cloud ClassificationDecoderFew-Shot 3D Point Cloud Classification+4Self-distillation Augmented Masked Autoencoders for Histopathological Image Classification
Self-supervised learning (SSL) has drawn increasing attention in histopathological image analysis in recent years. Compared to contrastive learning which is troubled with the false negative problem, i.e., semantically si…
Cell SegmentationClassificationContrastive LearningHistopathological Image Classification+5