Scaling Efficient Masked Image Modeling on Large Remote Sensing Dataset
Masked Image Modeling (MIM) has become an essential method for building foundational visual models in remote sensing (RS). However, the limitations in size and diversity of existing RS datasets restrict the ability of MIM methods to learn generalizable representations. Additionally, conventional MIM techniques, which require reconstructing all tokens, introduce unnecessary computational overhead. To address these issues, we present a new pre-training pipeline for RS models, featuring the creation of a large-scale RS dataset and an efficient MIM approach. We curated a high-quality dataset named OpticalRS-13M by collecting publicly available RS datasets and processing them through exclusion, slicing, and deduplication. OpticalRS-13M comprises 13 million optical images covering various RS tasks, such as object detection and pixel segmentation. To enhance efficiency, we propose SelectiveMAE, a pre-training method that dynamically encodes and reconstructs semantically rich patch tokens, thereby reducing the inefficiencies of traditional MIM models caused by redundant background pixels in RS images. Extensive experiments demonstrate that OpticalRS-13M significantly improves classification, detection, and segmentation performance, while SelectiveMAE increases training efficiency over 2 times. This highlights the effectiveness and scalability of our pipeline in developing RS foundational models.
Code (1)
Tasks
Aerial Scene ClassificationDiversityobject-detectionObject DetectionObject Detection In Aerial ImagesSemantic SegmentationSence ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Data Scaling in Masked Image Modeling
An important goal of self-supervised learning is to enable model pre-training to benefit from almost unlimited data. However, one method that has recently become popular, namely masked image modeling (MIM), is suspected …
Self-Supervised LearningRemoteCLIP: A Vision Language Foundation Model for Remote Sensing
General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation model…
ClassificationCross-Modal Retrievalimage-classificationImage Classification+8MIMRS: A Survey on Masked Image Modeling in Remote Sensing
Masked Image Modeling (MIM) is a self-supervised learning technique that involves masking portions of an image, such as pixels, patches, or latent representations, and training models to predict the missing information u…
Cloud RemovalSelf-Supervised LearningSuper-ResolutionSatMamba: Development of Foundation Models for Remote Sensing Imagery Using State Space Models
Foundation models refer to deep learning models pretrained on large unlabeled datasets through self-supervised algorithms. In the Earth science and remote sensing communities, there is growing interest in transforming th…
Earth ObservationState Space ModelsInteractive Masked Image Modeling for Multimodal Object Detection in Remote Sensing
Object detection in remote sensing imagery plays a vital role in various Earth observation applications. However, unlike object detection in natural scene images, this task is particularly challenging due to the abundanc…
Earth ObservationObjectobject-detectionObject Detection+1