DriveScape: High-Resolution Driving Video Generation by Multi-View Feature Fusion
Recent advancements in generative models offer promising solutions for synthesizing realistic driving videos, aiding in training autonomous driving perception models. However, existing methods often struggle with high-resolution multi-view generation, mainly due to the significant memory and computational overhead caused by simultaneously inputting multi-view videos into denoising diffusion models.In this paper, we propose a driving video generation framework based on multi-view feature fusion named DriveScape for multi-view 3D condition-guided video generation. We introduce a Bi-Directional Modulated Transformer (BiMoT) module to encode, fuse and inject multi-view features along with various 3D road structures and objects, which enables high-resolution multi-view generation. Consequently, our approach allows precise control over video generation, greatly enhancing realism and providing a robust solution for creating high-quality, multi-view driving videos.Our framework achieves state-of-the-art results on the nuScenes dataset, demonstrating impressive generative quality metrics with an FID score of 8.34 and an FVD score of 76.39, as well as superior performance across various perception tasks. This lays the foundation for more accurate environment simulation in autonomous driving. We plan to make our code and pre-trained model publicly available.Please refer to index.html webpage in the supplementary materials for more visualization results.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingDenoisingVideo GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches ofte…
Autonomous DrivingVideo GenerationInfinityDrive: Breaking Time Limits in Driving World Models
Autonomous driving systems struggle with complex scenarios due to limited access to diverse, extensive, and out-of-distribution driving data which are critical for safe navigation. World models offer a promising solution…
Autonomous DrivingDiversityVideo GenerationMyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control
High-quality driving video generation is crucial for providing training data for autonomous driving models. However, current generative models rarely focus on enhancing camera motion control under multi-view tasks, which…
Autonomous DrivingVideo GenerationPhysical Informed Driving World Model
Autonomous driving requires robust perception models trained on high-quality, large-scale multi-view driving videos for tasks like 3D object detection, segmentation and trajectory prediction. While world models provide a…
3D Object DetectionAutonomous Drivingmodelobject-detection+3MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is essential for applications like autonomous driving. However, existing methods are limi…
Autonomous DrivingVideo Generation