Learning Semantic-Aware Dynamics for Video Prediction
We propose an architecture and training scheme to predict video frames by explicitly modeling dis-occlusions and capturing the evolution of semantically consistent regions in the video. The scene layout (semantic map) and motion (optical flow) are decomposed into layers, which are predicted and fused with their context to generate future layouts and motions. The appearance of the scene is warped from past frames using the predicted motion in co-visible regions; dis-occluded regions are synthesized with content-aware inpainting utilizing the predicted scene layout. The result is a predictive model that explicitly represents objects and learns their class-specific motion, which we evaluate on video prediction benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Optical Flow EstimationPredictionVideo PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have attempted to mitigate this through self-supervised repres…
Representation LearningVideo PredictionDynaMind: Reconstructing Dynamic Visual Scenes from EEG by Aligning Temporal Dynamics and Multimodal Semantics to Guided Diffusion
Reconstruction dynamic visual scenes from electroencephalography (EEG) signals remains a primary challenge in brain decoding, limited by the low spatial resolution of EEG, a temporal mismatch between neural recordings an…
Video ReconstructionBrain DecodingFlowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videos
Prior works on 3D hand trajectory prediction are constrained by datasets that decouple motion from semantic supervision and by models that weakly link reasoning and action. To address these, we first present the EgoMAN d…
Trajectory PredictionUnbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasing
Video Scene Graph Generation (VidSGG) aims to capture dynamic relationships among entities by sequentially analyzing video frames and integrating visual and semantic information. However, VidSGG is challenged by signific…
Graph GenerationScene Graph GenerationTripletVideo scene graph generationSemantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis
Weather synthesis aims to add weather effects to input videos while preserving scene identity, structure, and motion. The key limitation of existing methods is the lack of diversity in weather appearance and effective co…
Semantic SegmentationAutonomous Driving