Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encoding
3D GAN inversion aims to achieve high reconstruction fidelity and reasonable 3D geometry simultaneously from a single image input. However, existing 3D GAN inversion methods rely on time-consuming optimization for each individual case. In this work, we introduce a novel encoder-based inversion framework based on EG3D, one of the most widely-used 3D GAN models. We leverage the inherent properties of EG3D's latent space to design a discriminator and a background depth regularization. This enables us to train a geometry-aware encoder capable of converting the input image into corresponding latent code. Additionally, we explore the feature space of EG3D and develop an adaptive refinement stage that improves the representation ability of features in EG3D to enhance the recovery of fine-grained textural details. Finally, we propose an occlusion-aware fusion operation to prevent distortion in unobserved regions. Our method achieves impressive results comparable to optimization-based methods while operating up to 500 times faster. Our framework is well-suited for applications such as semantic editing.
Code (0)
등록된 구현이 없습니다.
Tasks
3D geometrySimilar Papers 제목 키워드 기반
The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generation
Concept-bottleneck controllable generation routes multi-attribute control through a low-dimensional concept code that, at deployment, must be synthesised from a target attribute configuration. We study this problem in co…
Text GenerationModel Inversion Attacks against Graph Neural Networks
Many data mining tasks rely on graphs to model relational structures among individuals (nodes). Since relational data are often sensitive, there is an urgent need to evaluate the privacy risks in graph data. One famous p…
modelReinforcement Learning (RL)Differentially Private Data Generative Models
Deep neural networks (DNNs) have recently been widely adopted in various applications, and such success is largely due to a combination of algorithmic breakthroughs, computation resource improvements, and access to a lar…
BIG-bench Machine LearningFederated LearningInference AttackMembership Inference AttackAgainst All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks
Large Language Models (LLMs) are susceptible to malicious influence by cyber attackers through intrusions such as adversarial, backdoor, and embedding inversion attacks. In response, the burgeoning field of LLM Security …
AllALGEN: Few-shot Inversion Attacks on Textual Embeddings using Alignment and Generation
With the growing popularity of Large Language Models (LLMs) and vector databases, private textual data is increasingly processed and stored as numerical embeddings. However, recent studies have proven that such embedding…