Optical Flow Distillation: Towards Efficient and Stable Video Style Transfer
Video style transfer techniques inspire many exciting applications on mobile devices. However, their efficiency and stability are still far from satisfactory. To boost the transfer stability across frames, optical flow is widely adopted, despite its high computational complexity, e.g. occupying over 97% inference time. This paper proposes to learn a lightweight video style transfer network via knowledge distillation paradigm. We adopt two teacher networks, one of which takes optical flow during inference while the other does not. The output difference between these two teacher networks highlights the improvements made by optical flow, which is then adopted to distill the target student network. Furthermore, a low-rank distillation loss is employed to stabilize the output of student network by mimicking the rank of input videos. Extensive experiments demonstrate that our student network without an optical flow module is still able to generate stable video and runs much faster than the teacher network.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationOptical Flow EstimationStyle TransferVideo Style TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evolvement Constrained Adversarial Learning for Video Style Transfer
Video style transfer is a useful component for applications such as augmented reality, non-photorealistic rendering, and interactive games. Many existing methods use optical flow to preserve the temporal smoothness of th…
Generative Adversarial NetworkOptical Flow EstimationStyle TransferVideo Style TransferCharacterizing and Improving Stability in Neural Style Transfer
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to vid…
Optical Flow EstimationStyle TransferVideo Style TransferCreative Flow+ Dataset
We present the Creative Flow+ Dataset, the first diverse multi-style artistic video dataset richly labeled with per-pixel optical flow, occlusions, correspondences, segmentation labels, normals, and depth. Our dataset in…
3D Character Animation From A Single PhotoDepth EstimationImage AnimationObject Tracking+5Frame Difference-Based Temporal Loss for Video Stylization
Neural style transfer models have been used to stylize an ordinary video to specific styles. To ensure temporal inconsistency between the frames of the stylized video, a common approach is to estimate the optic flow of t…
Optical Flow EstimationStyle TransferSpatiotemporal Blind-Spot Network with Calibrated Flow Alignment for Self-Supervised Video Denoising
Self-supervised video denoising aims to remove noise from videos without relying on ground truth data, leveraging the video itself to recover clean frames. Existing methods often rely on simplistic feature stacking or ap…
DenoisingOptical Flow EstimationVideo Denoising