Fast Fourier Inception Networks for Occluded Video Prediction
Video prediction is a pixel-level task that generates future frames by employing the historical frames. There often exist continuous complex motions, such as object overlapping and scene occlusion in video, which poses great challenges to this task. Previous works either fail to well capture the long-term temporal dynamics or do not handle the occlusion masks. To address these issues, we develop the fully convolutional Fast Fourier Inception Networks for video prediction, termed \textit{FFINet}, which includes two primary components, \ie, the occlusion inpainter and the spatiotemporal translator. The former adopts the fast Fourier convolutions to enlarge the receptive field, such that the missing areas (occlusion) with complex geometric structures are filled by the inpainter. The latter employs the stacked Fourier transform inception module to learn the temporal evolution by group convolutions and the spatial movement by channel-wise Fourier convolutions, which captures both the local and the global spatiotemporal features. This encourages generating more realistic and high-quality future frames. To optimize the model, the recovery loss is imposed to the objective, \ie, minimizing the mean square error between the ground-truth frame and the recovery frame. Both quantitative and qualitative experimental results on five benchmarks, including Moving MNIST, TaxiBJ, Human3.6M, Caltech Pedestrian, and KTH, have demonstrated the superiority of the proposed approach. Our code is available at GitHub.
Code (1)
Tasks
PredictionVideo PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Inception-inspired LSTM for Next-frame Video Prediction
The problem of video frame prediction has received much interest due to its relevance to many computer vision applications such as autonomous vehicles or robotics. Supervised methods for video frame prediction rely on la…
Autonomous Vehiclesimage-classificationImage ClassificationPrediction+1A Comparison of Deep Learning Models for the Prediction of Hand Hygiene Videos
This paper presents a comparison of various deep learning models such as Exception, Resnet-50, and Inception V3 for the classification and prediction of hand hygiene gestures, which were recorded in accordance with the W…
GPUDisentangling Propagation and Generation for Video Prediction
A dynamic scene has two types of elements: those that move fluidly and can be predicted from previous frames, and those which are disoccluded (exposed) and cannot be extrapolated. Prior approaches to video prediction typ…
Predict Future Video FramesPredictionVideo PredictionAdversarial Video Generation on Complex Datasets
Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Advers…
3D Character Animation From A Single PhotoVideo GenerationVideo PredictionFlow and Depth Assisted Video Prediction with Latent Transformer
Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occl…
Video Prediction