SUV: Future Scene Understanding as Video Generation for End-to-End Driving
End-to-end driving requires a coherent understanding of future scenes, yet existing methods model these scenes using task-specific heads and output formats, with limited scalability. Can video generation instead provide a shared predictor? We introduce SUV, a unified end-to-end driving framework that casts future Scene Understanding as Video generation using a pretrained video foundation model. SUV models future appearance, semantics, relative depth, and instance-level dynamics as video streams with a shared video expert, without stream-specific visual prediction heads. Through joint video-action attention, the action expert attends to the latent representations of all future streams and generates the ego trajectory. Experiments show that SUV directly predicts all four future streams, while controlled ablations show that structured future supervision and direct future-stream access yield higher trajectory planning scores. With only a single front camera and no candidate-trajectory selection, SUV outperforms a broad set of recent state-of-the-art methods on both NAVSIM-v2 splits, achieving 91.0 EPDMS on navtest and 36.9 on navhard. On the long-tail WOD-E2E benchmark, SUV achieves a competitive RFS of 7.94.
Code (1)
Tasks
Scene UnderstandingTrajectory PlanningVideo GenerationSimilar Papers 제목 키워드 기반
LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving
Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use …
Instruction FollowingAutonomous DrivingVideo GenerationVideo PredictionHERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation
Driving World Models (DWMs) have become essential for autonomous driving by enabling future scene prediction. However, existing DWMs are limited to scene generation and fail to incorporate scene understanding, which invo…
Autonomous DrivingLanguage ModelingLanguage ModellingLarge Language Model+3MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer
Urban scene synthesis with video generation models has recently shown great potential for autonomous driving. Existing video generation approaches to autonomous driving primarily focus on RGB video generation and lack th…
Scene UnderstandingAutonomous DrivingVideo GenerationStag-1: Towards Realistic 4D Driving Simulation with Video Generation Model
4D driving simulation is essential for developing realistic autonomous driving simulators. Despite advancements in existing methods for generating driving scenes, significant challenges remain in view transformation and …
Autonomous DrivingAutonomous VehiclesScene UnderstandingVideo GenerationUniDrive-WM: Unified Understanding, Planning and Generation World Model for Autonomous Driving
World models have become central to autonomous driving, where accurate scene understanding and future prediction are crucial for safe control. Recent work has explored using vision-language models (VLMs) for planning, ye…
Scene UnderstandingTrajectory PlanningAutonomous DrivingImage Generation