Lightweight Multiplane Images Network for Real-Time Stereoscopic Conversion from Planar Video
With the rapid development of stereoscopic display technologies, especially glasses-free 3D screens, and virtual reality devices, stereoscopic conversion has become an important task to address the lack of high-quality stereoscopic image and video resources. Current stereoscopic conversion algorithms typically struggle to balance reconstruction performance and inference efficiency. This paper proposes a planar video real-time stereoscopic conversion network based on multi-plane images (MPI), which consists of a detail branch for generating MPI and a depth-semantic branch for perceiving depth information. Unlike models that depend on explicit depth map inputs, the proposed method employs a lightweight depth-semantic branch to extract depth-aware features implicitly. To optimize the lightweight branch, a heavy training but light inference strategy is adopted, which involves designing a coarse-to-fine auxiliary branch that is only used during the training stage. In addition, the proposed method simplifies the MPI rendering process for stereoscopic conversion scenarios to further accelerate the inference. Experimental results demonstrate that the proposed method can achieve comparable performance to some state-of-the-art (SOTA) models and support real-time inference at 2K resolution. Compared to the SOTA TMPI algorithm, the proposed method obtains similar subjective quality while achieving over $40\times$ inference acceleration.
Code (0)
등록된 구현이 없습니다.
Tasks
2kSimilar Papers 제목 키워드 기반
Adaptive Multiplane Image Generation from a Single Internet Picture
In the last few years, several works have tackled the problem of novel view synthesis from stereo images or even from a single picture. However, previous methods are computationally expensive, specially for high-resoluti…
Depth EstimationImage GenerationKnowledge DistillationNovel View SynthesisMultiPlaneNeRF: Neural Radiance Field with Non-Trainable Representation
NeRF is a popular model that efficiently represents 3D objects from 2D images. However, vanilla NeRF has some important limitations. NeRF must be trained on each object separately. The training time is long since we enco…
DecoderNeRFSinMPI: Novel View Synthesis from a Single Image with Expanded Multiplane Images
Single-image novel view synthesis is a challenging and ongoing problem that aims to generate an infinite number of consistent views from a single input image. Although significant efforts have been made to advance the qu…
Novel View SynthesisReal-Time Neural Character Rendering with Pose-Guided Multiplane Images
We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the dr…
Image-to-Image TranslationNeural RenderingNovel View SynthesisRemote Sensing Novel View Synthesis with Implicit Multiplane Representations
Novel view synthesis of remote sensing scenes is of great significance for scene visualization, human-computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photog…
Novel View Synthesis