NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
This paper presents a unified multimodal pre-trained model called N\"UWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate N\"UWA on 8 downstream tasks. Compared to several strong baselines, N\"UWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.
Code (1)
Tasks
DecoderImage GenerationText to Image GenerationText-to-Image GenerationText-to-Video GenerationVideo GenerationVideo PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It …
Generative Adversarial NetworkNeural RenderingTranslationBizGenEval: A Systematic Benchmark for Commercial Visual Content Generation
Recent advances in image generation models have expanded their applications beyond aesthetic imagery toward practical visual content creation. However, existing benchmarks mainly focus on natural image synthesis and fail…
Image GenerationPARASOL: Parametric Style Control for Diffusion Image Synthesis
We propose PARASOL, a multi-modal synthesis model that enables disentangled, parametric control of the visual style of the image by jointly conditioning synthesis on both content and a fine-grained visual style embedding…
Image GenerationEffective Training Data Synthesis for Improving MLLM Chart Understanding
Being able to effectively read scientific plots, or chart understanding, is a central part toward building effective agents for science. However, existing multimodal large language models (MLLMs), especially open-source …
Exemplar-based Pattern Synthesis with Implicit Periodic Field Network
Synthesis of ergodic, stationary visual patterns is widely applicable in texturing, shape modeling, and digital content creation. The wide applicability of this technique thus requires the pattern synthesis approaches to…
Generative Adversarial NetworkTexture Synthesis