Neural Face Video Compression using Multiple Views
Recent advances in deep generative models led to the development of neural face video compression codecs that use an order of magnitude less bandwidth than engineered codecs. These neural codecs reconstruct the current frame by warping a source frame and using a generative model to compensate for imperfections in the warped source frame. Thereby, the warp is encoded and transmitted using a small number of keypoints rather than a dense flow field, which leads to massive savings compared to traditional codecs. However, by relying on a single source frame only, these methods lead to inaccurate reconstructions (e.g. one side of the head becomes unoccluded when turning the head and has to be synthesized). Here, we aim to tackle this issue by relying on multiple source frames (views of the face) and present encouraging results.
Code (0)
등록된 구현이 없습니다.
Tasks
Video CompressionSimilar Papers 제목 키워드 기반
Perceptual Quality Assessment of Face Video Compression: A Benchmark and An Effective Method
Recent years have witnessed an exponential increase in the demand for face video compression, and the success of artificial intelligence has expanded the boundaries beyond traditional hybrid video coding. Generative codi…
Video CompressionVideo Quality AssessmentVisual Question Answering (VQA)Multi-modality Deep Restoration of Extremely Compressed Face Videos
Arguably the most common and salient object in daily video communications is the talking head, as encountered in social media, virtual classrooms, teleconferences, news broadcasting, talk shows, etc. When communication b…
QuantizationOrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis
Novel View Synthesis (NVS) aims to generate unseen views of a 3D object given a limited number of known views. Existing methods often struggle to synthesize plausible views for unobserved regions, particularly under sing…
Novel View SynthesisVideo GenerationCompressing Video Calls using Synthetic Talking Heads
We leverage the modern advancements in talking head generation to propose an end-to-end system for talking head video compression. Our algorithm transmits pivot frames intermittently while the rest of the talking head vi…
Face ReenactmentTalking Head GenerationVideo CompressionLow-Latency Neural Stereo Streaming
The rise of new video modalities like virtual reality or autonomous driving has increased the demand for efficient multi-view video compression methods, both in terms of rate-distortion (R-D) performance and in terms of …
Autonomous DrivingMotion CompensationVideo Compression