Online Video Super-Resolution with Convolutional Kernel Bypass Graft
Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider the distortion quality and ignore crucial requirements for online applications, e.g., low latency and low model complexity. In this paper, we focus on online video transmission, in which VSR algorithms are required to generate high-resolution video sequences frame by frame in real time. To address such challenges, we propose an extremely low-latency VSR algorithm based on a novel kernel knowledge transfer method, named convolutional kernel bypass graft (CKBG). First, we design a lightweight network structure that does not require future frames as inputs and saves extra time costs for caching these frames. Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with ``kernel grafts'', which are extra convolutional kernels containing the prior knowledge of external pretrained image SR models. In the testing phase, we further accelerate the grafted multi-branch network by converting it into a simple single-path structure. Experiment results show that our proposed method can process online video sequences up to 110 FPS, with very low model complexity and competitive SR performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Super-ResolutionTransfer LearningVideo Super-ResolutionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Deformable Kernel Convolutional Network for Video Extreme Super-Resolution
Video super-resolution, which attempts to reconstruct high-resolution video frames from their corresponding low-resolution versions, has received increasingly more attention in recent years. Most existing approaches opt …
Super-ResolutionVideo Super-ResolutionDeep Blind Video Super-resolution
Existing video super-resolution (SR) algorithms usually assume that the blur kernels in the degradation process are known and do not model the blur kernels in the restoration. However, this assumption does not hold for v…
Image DeconvolutionImage RestorationMotion EstimationSuper-Resolution+1Online Streaming Video Super-Resolution with Convolutional Look-Up Table
Online video streaming has fundamental limitations on the transmission bandwidth and computational capacity and super-resolution is a promising potential solution. However, applying existing video super-resolution method…
Super-ResolutionVideo Super-ResolutionTemporal Kernel Consistency for Blind Video Super-Resolution
Deep learning-based blind super-resolution (SR) methods have recently achieved unprecedented performance in upscaling frames with unknown degradation. These models are able to accurately estimate the unknown downscaling …
Blind Super-ResolutionSuper-ResolutionVideo Super-ResolutionSelf-Supervised Deep Blind Video Super-Resolution
Existing deep learning-based video super-resolution (SR) methods usually depend on the supervised learning approach, where the training data is usually generated by the blurring operation with known or predefined kernels…
Optical Flow EstimationSelf-Supervised LearningSuper-ResolutionVideo Restoration+1