SenPa-MAE: Sensor Parameter Aware Masked Autoencoder for Multi-Satellite Self-Supervised Pretraining
This paper introduces SenPa-MAE, a transformer architecture that encodes the sensor parameters of an observed multispectral signal into the image embeddings. SenPa-MAE can be pre-trained on imagery of different satellites with non-matching spectral or geometrical sensor characteristics. To incorporate sensor parameters, we propose a versatile sensor parameter encoding module as well as a data augmentation strategy for the diversification of the pre-training dataset. This enables the model to effectively differentiate between various sensors and gain an understanding of sensor parameters and the correlation to the observed signal. Given the rising number of Earth observation satellite missions and the diversity in their sensor specifications, our approach paves the way towards a sensor-independent Earth observation foundation model. This opens up possibilities such as cross-sensor training and sensor-independent inference.
Code (1)
Tasks
Data AugmentationDiversityEarth ObservationSimilar Papers 제목 키워드 기반
Exploring Masked Autoencoders for Sensor-Agnostic Image Retrieval in Remote Sensing
Self-supervised learning through masked autoencoders (MAEs) has recently attracted great attention for remote sensing (RS) image representation learning, and thus embodies a significant potential for content-based image …
Content-Based Image RetrievalImage RetrievalRepresentation LearningRetrieval+1MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning
With the exponential growth of multimedia data, leveraging multimodal sensors presents a promising approach for improving accuracy in human activity recognition. Nevertheless, accurately identifying these activities usin…
Activity RecognitionHuman Activity RecognitionOne-Shot LearningLightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observation
Recent advances in Earth Observation have focused on large-scale foundation models. However, these models are computationally expensive, limiting their accessibility and reuse for downstream tasks. In this work, we inves…
MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition
Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performa…
Self-Supervised LearningSign Language RecognitionHiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series
Wearable sensors provide abundant physiological time series, yet the principles governing their predictive utility remain unclear. We hypothesize that temporal resolution is a fundamental axis of representation learning,…
Representation Learning