Context Autoencoder for Self-Supervised Representation Learning
We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches. The network is an encoder-regressor-decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://github.com/Atten4Vis/CAE.
Code (6)
Tasks
DecoderInstance Segmentationobject-detectionObject DetectionRepresentation LearningSelf-Supervised Image ClassificationSelf-Supervised LearningSemantic SegmentationSimilar Papers 제목 키워드 기반
Boarding for ISS: Imbalanced Self-Supervised: Discovery of a Scaled Autoencoder for Mixed Tabular Datasets
The field of imbalanced self-supervised learning, especially in the context of tabular data, has not been extensively studied. Existing research has predominantly focused on image datasets. This paper aims to fill this g…
Dimensionality ReductionSelf-Supervised LearningDeep Anomaly Detection in Text
Deep anomaly detection methods have become increasingly popular in recent years, with methods like Stacked Autoencoders, Variational Autoencoders, and Generative Adversarial Networks greatly improving the state-of-the-ar…
Anomaly DetectionRepresentation LearningSelf-Supervised LearningUnsupervised Anomaly DetectionMAESIL: Masked Autoencoder for Enhanced Self-supervised Medical Image Learning
Training deep learning models for three-dimensional (3D) medical imaging, such as Computed Tomography (CT), is fundamentally challenged by the scarcity of labeled data. While pre-training on natural images is common, it …
Self-Supervised LearningComputational EfficiencyMasked Autoencoders in 3D Point Cloud Representation Learning
Transformer-based Self-supervised Representation Learning methods learn generic features from unlabeled datasets for providing useful network initialization parameters for downstream tasks. Recently, self-supervised lear…
Point Cloud CompletionPoint cloud reconstructionRepresentation LearningSelf-Supervised LearningLLM-based Contrastive Self-Supervised AMR Learning with Masked Graph Autoencoders for Fake News Detection
The proliferation of misinformation in the digital age has led to significant societal challenges. Existing approaches often struggle with capturing long-range dependencies, complex semantic relations, and the social dyn…
Fake News Detection