Learning Modulated Transformation in GANs
The success of style-based generators largely benefits from style modulation, which helps take care of the cross-instance variation within data. However, the instance-wise stochasticity is typically introduced via regular convolution, where kernels interact with features at some fixed locations, limiting its capacity for modeling geometric variation. To alleviate this problem, we equip the generator in generative adversarial networks (GANs) with a plug-and-play module, termed as modulated transformation module (MTM). This module predicts spatial offsets under the control of latent codes, based on which the convolution operation can be applied at variable locations for different instances, and hence offers the model an additional degree of freedom to handle geometry deformation. Extensive experiments suggest that our approach can be faithfully generalized to various generative tasks, including image generation, 3D-aware image synthesis, and video generation, and get compatible with state-of-the-art frameworks without any hyper-parameter tuning. It is noteworthy that, towards human generation on the challenging TaiChi dataset, we improve the FID of StyleGAN3 from 21.36 to 13.60, demonstrating the efficacy of learning modulated geometry transformation. Code and models are available at https://github.com/limbo0000/mtm.Submission Number: 3123
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Co-modulated GANs link image-conditional GANs and unconditional modulated models to address large-scale image completion tasks. Co-modulation brings stochastic and conditional style representations together. To improve e…
Image-to-Image TranslationTranslationIterGANs: Iterative GANs to Learn and Control 3D Object Transformation
We are interested in learning visual representations which allow for 3D manipulations of visual objects based on a single 2D image. We cast this into an image-to-image transformation task, and propose Iterative Generativ…
ObjectNICE: NoIse-modulated Consistency rEgularization for Data-Efficient GANs
Generative Adversarial Networks (GANs) are powerful tools for image synthesis. However, they require access to vast amounts of training data, which is often costly and prohibitive. Limited data affects GANs, leading to d…
Segmentation of Head and Neck Organs at Risk Using CNN with Batch Dice Loss
This paper deals with segmentation of organs at risk (OAR) in head and neck area in CT images which is a crucial step for reliable intensity modulated radiotherapy treatment. We introduce a convolution neural network wit…
DecoderSegmentationVisually Evaluating Generative Adversarial Networks Using Itself under Multivariate Time Series
Visually evaluating the goodness of generated Multivariate Time Series (MTS) are difficult to implement, especially in the case that the generative model is Generative Adversarial Networks (GANs). We present a general fr…
Time SeriesTime Series Analysis