Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
We present DeepEarth, a self-supervised multi-modal world model with Earth4D, a novel planetary-scale 4D space-time positional encoder. Earth4D extends 3D multi-resolution hash encoding to include time, efficiently scaling across the planet over centuries with sub-meter, sub-second precision. Multi-modal encoders (e.g. vision-language models) are fused with Earth4D embeddings and trained via masked reconstruction. We demonstrate Earth4D's expressive power by achieving state-of-the-art performance on an ecological forecasting benchmark. Earth4D with learnable hash probing surpasses a multi-modal foundation model pre-trained on substantially more data. Access open source code and download models at: https://github.com/legel/deepearth
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Self-Supervised Enhancement of Forward-Looking Sonar Images: Bridging Cross-Modal Degradation Gaps through Feature Space Transformation and Multi-Frame Fusion
Enhancing forward-looking sonar images is critical for accurate underwater target detection. Current deep learning methods mainly rely on supervised training with simulated data, but the difficulty in obtaining high-qual…
Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition
Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when desi…
Emotion ClassificationEmotion Recognition in ConversationFacial Expression RecognitionSelf-Supervised LearningBEVWorld: A Multimodal World Model for Autonomous Driving via Unified BEV Latent Space
World models are receiving increasing attention in autonomous driving for their ability to predict potential future scenarios. In this paper, we present BEVWorld, a novel approach that tokenizes multimodal sensor inputs …
Autonomous DrivingDecodermotion predictionMulti-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning
This work focuses on learning useful and robust deep world models using multiple, possibly unreliable, sensors. We find that current methods do not sufficiently encourage a shared representation between modalities; this …
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning (RL)Self-Supervised Multimodal NeRF for Autonomous Driving
In this paper, we propose a Neural Radiance Fields (NeRF) based framework, referred to as Novel View Synthesis Framework (NVSF). It jointly learns the implicit neural representation of space and time-varying scene for bo…
Autonomous DrivingNeRFNovel View Synthesis