Back to the Features: DINO as a Foundation for Video World Models
We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.
Code (0)
등록된 구현이 없습니다.
Tasks
Video PredictionSimilar Papers 제목 키워드 기반
DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models
We propose DINOBot, a novel imitation learning framework for robot manipulation, which leverages the image-level and pixel-level capabilities of features extracted from Vision Transformers trained with DINO. When interac…
Imitation LearningObjectRetrievalRobot ManipulationDriving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
Driven by the emergence of Controllable Video Diffusion, existing Sim2Real methods for autonomous driving video generation typically rely on explicit intermediate representations to bridge the domain gap. However, these …
Dimensionality ReductionAutonomous DrivingVideo GenerationDINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification
Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work propose…
Image ClassificationDino U-Net: Exploiting High-Fidelity Dense Features from Foundation Models for Medical Image Segmentation
Foundation models pre-trained on large-scale natural image datasets offer a powerful paradigm for medical image segmentation. However, effectively transferring their learned representations for precise clinical applicati…
Medical Image SegmentationDimensionality ReductionDINO-Mix: Enhancing Visual Place Recognition with Foundational Vision Model and Feature Mixing
Utilizing visual place recognition (VPR) technology to ascertain the geographical location of publicly available images is a pressing issue for real-world VPR applications. Although most current VPR methods achieve favor…
Visual Place Recognition