Physics-informed Guided Disentanglement in Generative Networks
Image-to-image translation (i2i) networks suffer from entanglement effects in presence of physics-related phenomena in target domain (such as occlusions, fog, etc), lowering altogether the translation quality, controllability and variability. In this paper, we propose a general framework to disentangle visual traits in target images. Primarily, we build upon collection of simple physics models, guiding the disentanglement with a physical model that renders some of the target traits, and learning the remaining ones. Because physics allows explicit and interpretable outputs, our physical models (optimally regressed on target) allows generating unseen scenarios in a controllable manner. Secondarily, we show the versatility of our framework to neural-guided disentanglement where a generative network is used in place of a physical model in case the latter is not directly accessible. Altogether, we introduce three strategies of disentanglement being guided from either a fully differentiable physics model, a (partially) non-differentiable physics model, or a neural network. The results show our disentanglement strategies dramatically increase performances qualitatively and quantitatively in several challenging scenarios for image translation.
Code (1)
Tasks
DisentanglementImage-to-Image TranslationTranslationSimilar Papers 제목 키워드 기반
Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery
We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual …
DecoderDisentanglementscientific discoveryVariational InferenceA Residual Guided strategy with Generative Adversarial Networks in training Physics-Informed Transformer Networks
Nonlinear partial differential equations (PDEs) are pivotal in modeling complex physical systems, yet traditional Physics-Informed Neural Networks (PINNs) often struggle with unresolved residuals in critical spatiotempor…
Physics-guided generative adversarial network to learn physical models
This short note describes the concept of guided training of deep neural networks (DNNs) to learn physically reasonable solutions. DNNs are being widely used to predict phenomena in physics and mechanics. One of the issue…
Generative Adversarial NetworkA Unified Generative-Predictive Framework for Deterministic Inverse Design
Inverse design of heterogeneous material microstructures is a fundamentally ill-posed and famously computationally expensive problem. This is exacerbated by the high-dimensional design spaces associated with finely resol…
CryoGEM: Physics-Informed Generative Cryo-Electron Microscopy
In the past decade, deep conditional generative models have revolutionized the generation of realistic images, extending their application from entertainment to scientific domains. Single-particle cryo-electron microscop…
Contrastive LearningPose EstimationTranslation