VaViM and VaVAM: Autonomous Driving through Video Generative Modeling
We explore the potential of large-scale generative video models for autonomous driving, introducing an open-source auto-regressive video model (VaViM) and its companion video-action model (VaVAM) to investigate how video pre-training transfers to real-world driving. VaViM is a simple auto-regressive video model that predicts frames using spatio-temporal token sequences. We show that it captures the semantics and dynamics of driving scenes. VaVAM, the video-action model, leverages the learned representations of VaViM to generate driving trajectories through imitation learning. Together, the models form a complete perception-to-action pipeline. We evaluate our models in open- and closed-loop driving scenarios, revealing that video-based pre-training holds promise for autonomous driving. Key insights include the semantic richness of the learned representations, the benefits of scaling for video synthesis, and the complex relationship between model size, data, and safety metrics in closed-loop evaluations. We release code and model weights at https://github.com/valeoai/VideoActionModel
Code (1)
Tasks
Autonomous DrivingImitation LearningSimilar Papers 제목 키워드 기반
DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving
Pretrained foundation models have become an important basis for end-to-end autonomous driving. In contrast to vision-language models pretrained primarily on static image-text pairs, video generative models capture tempor…
Scene UnderstandingAutonomous DrivingMagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is essential for applications like autonomous driving. However, existing methods are limi…
Autonomous DrivingVideo GenerationInstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video Generation
Autonomous driving relies on robust models trained on high-quality, large-scale multi-view driving videos. While world models offer a cost-effective solution for generating realistic driving videos, they struggle to main…
Autonomous DrivingSpatial ReasoningVideo GenerationDriving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
In autonomous driving, predicting future events in advance and evaluating the foreseeable risks empowers autonomous vehicles to better plan their actions, enhancing safety and efficiency on the road. To this end, we prop…
Autonomous DrivingAutonomous VehiclesHybrid Video Anomaly Detection for Anomalous Scenarios in Autonomous Driving
In autonomous driving, the most challenging scenarios can only be detected within their temporal context. Most video anomaly detection approaches focus either on surveillance or traffic accidents, which are only a subfie…
Anomaly DetectionAutonomous DrivingVideo Anomaly Detection