Stochastic Video Prediction with Structure and Motion
While stochastic video prediction models enable future prediction under uncertainty, they mostly fail to model the complex dynamics of real-world scenes. For example, they cannot provide reliable predictions for scenes with a moving camera and independently moving foreground objects in driving scenarios. The existing methods fail to fully capture the dynamics of the structured world by only focusing on changes in pixels. In this paper, we assume that there is an underlying process creating observations in a video and propose to factorize it into static and dynamic components. We model the static part based on the scene structure and the ego-motion of the vehicle, and the dynamic part based on the remaining motion of the dynamic objects. By learning separate distributions of changes in foreground and background, we can decompose the scene into static and dynamic parts and separately model the change in each. Our experiments demonstrate that disentangling structure and motion helps stochastic video prediction, leading to better future predictions in complex driving scenarios on two real-world driving datasets, KITTI and Cityscapes.
Code (0)
등록된 구현이 없습니다.
Tasks
Future predictionPredictionVideo PredictionSimilar Papers 제목 키워드 기반
State-space Decomposition Model for Video Prediction Considering Long-term Motion Trend
Stochastic video prediction enables the consideration of uncertainty in future motion, thereby providing a better reflection of the dynamic nature of the environment. Stochastic video prediction methods based on image au…
motion predictionPredictionState Space ModelsVideo PredictionSLAMP: Stochastic Latent Appearance and Motion Prediction
Motion is an important cue for video prediction and often utilized by separating video content into static and dynamic components. Most of the previous work utilizing motion is deterministic but there are stochastic meth…
Autonomous Drivingmotion predictionPredictionVideo Generation+1STDiff: Spatio-temporal Diffusion for Continuous Stochastic Video Prediction
Predicting future frames of a video is challenging because it is difficult to learn the uncertainty of the underlying factors influencing their contents. In this paper, we propose a novel video prediction model, which ha…
PredictionVideo PredictionInference-time Stochastic Refinement of GRU-Normalizing Flow for Real-time Video Motion Transfer
Real-time video motion transfer applications such as immersive gaming and vision-based anomaly detection require accurate yet diverse future predictions to support realistic synthesis and robust downstream decision makin…
Time Series ForecastingAnomaly DetectionDecision MakingUnsupervised Learning of Object Structure and Dynamics from Videos
Extracting and predicting object structure and dynamics from videos without supervision is a major challenge in machine learning. To address this challenge, we adopt a keypoint-based image representation and learn a stoc…
Action Recognitioncontinuous-controlContinuous ControlObject+2