Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latant Space
Advanced end-to-end autonomous driving systems predict other vehicles' motions and plan ego vehicle's trajectory. The world model that can foresee the outcome of the trajectory has been used to evaluate the end-to-end autonomous driving system. However, existing world models predominantly emphasize the trajectory of the ego vehicle and leave other vehicles uncontrollable. This limitation hinders their ability to realistically simulate the interaction between the ego vehicle and the driving scenario. In addition, it remains a challenge to match multiple trajectories with each vehicle in the video to control the video generation. To address above issues, a driving \textbf{W}orld \textbf{M}odel named EOT-WM is proposed in this paper, unifying \textbf{E}go-\textbf{O}ther vehicle \textbf{T}rajectories in videos. Specifically, we first project ego and other vehicle trajectories in the BEV space into the image coordinate to match each trajectory with its corresponding vehicle in the video. Then, trajectory videos are encoded by the Spatial-Temporal Variational Auto Encoder to align with driving video latents spatially and temporally in the unified visual space. A trajectory-injected diffusion Transformer is further designed to denoise the noisy video latents for video generation with the guidance of ego-other vehicle trajectories. In addition, we propose a metric based on control latent similarity to evaluate the controllability of trajectories. Extensive experiments are conducted on the nuScenes dataset, and the proposed model outperforms the state-of-the-art method by 30\% in FID and 55\% in FVD. The model can also predict unseen driving scenes with self-produced trajectories.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingVideo GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning
Autonomous vehicles (AVs) are on the road. To safely and efficiently interact with other road participants, AVs have to accurately predict the behavior of surrounding vehicles and plan accordingly. Such prediction should…
Autonomous VehiclesPredictionreinforcement-learningReinforcement Learning+1Probabilistic Prediction of Vehicle Semantic Intention and Motion
Accurately predicting the possible behaviors of traffic participants is an essential capability for future autonomous vehicles. The majority of current researches fix the number of driving intentions by considering only …
Autonomous Vehiclesmotion predictionPredictionStochastic Time-Optimal Trajectory Planning for Connected and Automated Vehicles in Mixed-Traffic Merging Scenarios
Addressing safe and efficient interaction between connected and automated vehicles (CAVs) and human-driven vehicles in a mixed-traffic environment has attracted considerable attention. In this paper, we develop a framewo…
regressionTrajectory PlanningUncertainty QuantificationExtraction and Assessment of Naturalistic Human Driving Trajectories from Infrastructure Camera and Radar Sensors
Collecting realistic driving trajectories is crucial for training machine learning models that imitate human driving behavior. Most of today's autonomous driving datasets contain only a few trajectories per location and …
Autonomous DrivingSensor FusionSafe Real-World Autonomous Driving by Learning to Predict and Plan with a Mixture of Experts
The goal of autonomous vehicles is to navigate public roads safely and comfortably. To enforce safety, traditional planning approaches rely on handcrafted rules to generate trajectories. Machine learning-based systems, o…
Autonomous DrivingAutonomous VehiclesMixture-of-ExpertsNavigate