Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
Controllable video generation has attracted significant attention, largely due to advances in video diffusion models. In domains such as autonomous driving, it is essential to develop highly accurate predictions for object motions. This paper tackles a crucial challenge of how to exert precise control over object motion for realistic video synthesis. To accomplish this, we 1) control object movements using bounding boxes and extend this control to the renderings of 2D or 3D boxes in pixel space, 2) employ a distinct, specialized model to forecast the trajectories of object bounding boxes based on their previous and, if desired, future positions, and 3) adapt and enhance a separate video diffusion network to create video content based on these high quality trajectory forecasts. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation. Extensive experiments conducted on the KITTI, Virtual-KITTI 2, BDD100k, and nuScenes datasets validate the effectiveness of our approach in producing realistic and controllable video generation.
Code (1)
Tasks
Autonomous DrivingObjectVideo GenerationVideo PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Ctrl-Crash: Controllable Diffusion for Realistic Car Crashes
Video diffusion techniques have advanced significantly in recent years; however, they struggle to generate realistic imagery of car crashes due to the scarcity of accident events in most driving datasets. Improving traff…
counterfactualVideo GenerationPhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
Existing video generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability. To overcome these limitations, we introduce PhysCtrl, a novel f…
Video GenerationEditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing
High-fidelity generative video editing has seen significant quality improvements by leveraging pre-trained video foundation models. However, their computational cost is a major bottleneck, as they are often designed to i…
Video InpaintingCtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
We tackle the dual challenges of video understanding and controllable video generation within a unified diffusion framework. Our key insights are two-fold: geometry-only cues (e.g., depth, edges) are insufficient: they s…
Video GenerationReliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning
With the rapid advancement of vision-language models, an increasing number of studies have explored their potential for SVG generation tasks. Although existing approaches improve performance by constructing large-scale S…
Reinforcement Learning