360VFI: A Dataset and Benchmark for Omnidirectional Video Frame Interpolation
Head-mounted 360{\deg} displays and portable 360{\deg} cameras have significantly progressed, providing viewers a realistic and immersive experience. However, many omnidirectional videos have low frame rates that can lead to visual fatigue, and the prevailing plane frame interpolation methodologies are unsuitable for omnidirectional video interpolation because they are designed solely for traditional videos. This paper introduces the benchmark dataset, 360VFI, for Omnidirectional Video Frame Interpolation. We present a practical implementation that introduces a distortion prior from omnidirectional video into the network to modulate distortions. Specifically, we propose a pyramid distortion-sensitive feature extractor that uses the unique characteristics of equirectangular projection (ERP) format as prior information. Moreover, we devise a decoder that uses an affine transformation to further facilitate the synthesis of intermediate frames. 360VFI is the first dataset and benchmark that explores the challenge of Omnidirectional Video Frame Interpolation. Through our benchmark analysis, we present four different distortion condition scenes in the proposed 360VFI dataset to evaluate the challenges triggered by distortion during interpolation. Besides, experimental results demonstrate that Omnidirectional Video Interpolation can be effectively improved by modeling for omnidirectional distortion.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderERPVideo Frame InterpolationSimilar Papers 제목 키워드 기반
Novel tile segmentation scheme for omnidirectional video
Regular omnidirectional video encoding technics use map projection to flatten a scene from a spherical shape into one or several 2D shapes. Common projection methods including equirectangular and cubic projection have va…
Video SegmentationVideo Semantic SegmentationLearning Omnidirectional Flow in 360-degree Video via Siamese Representation
Optical flow estimation in omnidirectional videos faces two significant issues: the lack of benchmark datasets and the challenge of adapting perspective video-based methods to accommodate the omnidirectional nature. This…
DiversityOptical Flow EstimationRepresentation LearningOptical Flow Estimation in 360$^\circ$ Videos: Dataset, Model and Application
Optical flow estimation has been a long-lasting and fundamental problem in the computer vision community. However, despite the advances of optical flow estimation in perspective videos, the 360$^\circ$ videos counterpart…
Activity RecognitionEgocentric Activity RecognitionOptical Flow EstimationRepresentation LearningAudio-visual Saliency for Omnidirectional Videos
Visual saliency prediction for omnidirectional videos (ODVs) has shown great significance and necessity for omnidirectional videos to help ODV coding, ODV transmission, ODV rendering, etc.. However, most studies only con…
PredictionSaliency Prediction360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos
Visual object tracking and segmentation in omnidirectional videos are challenging due to the wide field-of-view and large spherical distortion brought by 360{\deg} images. To alleviate these problems, we introduce a nove…
ObjectObject TrackingSegmentationSemantic Segmentation+3