Intrinsic Temporal Regularization for High-resolution Human Video Synthesis
Temporal consistency is crucial for extending image processing pipelines to the video domain, which is often enforced with flow-based warping error over adjacent frames. Yet for human video synthesis, such scheme is less reliable due to the misalignment between source and target video as well as the difficulty in accurate flow estimation. In this paper, we propose an effective intrinsic temporal regularization scheme to mitigate these issues, where an intrinsic confidence map is estimated via the frame generator to regulate motion estimation via temporal loss modulation. This creates a shortcut for back-propagating temporal loss gradients directly to the front-end motion estimator, thus improving training stability and temporal coherence in output videos. We apply our intrinsic temporal regulation to single-image generator, leading to a powerful "INTERnet" capable of generating $512\times512$ resolution human action videos with temporal-coherent, realistic visual details. Extensive experiments demonstrate the superiority of proposed INTERnet over several competitive baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
Motion EstimationVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
EEG-based Graph-guided Domain Adaptation for Robust Cross-Session Emotion Recognition
Accurate recognition of human emotional states is critical for effective human-machine interaction. Electroencephalography (EEG) offers a reliable source for emotion recognition due to its high temporal resolution and it…
Emotion RecognitionDomain AdaptationFISR: Deep Joint Frame Interpolation and Super-Resolution with a Multi-scale Temporal Loss
Super-resolution (SR) has been widely used to convert low-resolution legacy videos to high-resolution (HR) ones, to suit the increasing resolution of displays (e.g. UHD TVs). However, it becomes easier for humans to noti…
2k4k8kSpace-time Video Super-resolution+3Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization
Spatiotemporal modeling is critical for understanding complex systems across various scientific and engineering disciplines, but governing equations are often not fully known or computationally intractable due to inheren…
Operator learningSpatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild
Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for moving points captured in multiple unsy…
3D Human ReconstructionDepth Superresolution using Motion Adaptive Regularization
Spatial resolution of depth sensors is often significantly lower compared to that of conventional optical cameras. Recent work has explored the idea of improving the resolution of depth using higher resolution intensity …