Neural Rerendering in the Wild
We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos.
Code (0)
등록된 구현이 없습니다.
Tasks
3D ReconstructionSimilar Papers 제목 키워드 기반
World in World: Explore the World with World Models
Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised…
Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool
We present a neural rendering framework that maps a voxelized scene into a high quality image. Highly-textured objects and scene element interactions are realistically rendered by our method, despite having a rough repre…
Image GenerationNeural RenderingLatency Reduction in CloudVR: Cloud Prediction, Edge Correction
Current virtual reality (VR) headsets encounter a trade-off between high processing power and affordability. Consequently, offloading 3D rendering to remote servers helps reduce costs, battery usage, and headset weight. …
PredictionIRAD: Implicit Representation-driven Image Resampling against Adversarial Attacks
We introduce a novel approach to counter adversarial attacks, namely, image resampling. Image resampling transforms a discrete image into a new one, simulating the process of scene recapturing or rerendering as specified…
Adversarial RobustnessAI-based Wildfire Prevention, Detection and Suppression System
Wildfires pose a serious threat to the environment of the world. The global wildfire season length has increased by 19% and severe wildfires have besieged nations around the world. Every year, forests are burned by wildf…