Exploring Vision Transformers as Diffusion Learners

2022-12-28 · He Cao, Jianan Wang, Tianhe Ren, Xianbiao Qi, Yihao Chen, Yuan YAO, Lei Zhang

Score-based diffusion models have captured widespread attention and funded fast progress of recent vision generative tasks. In this paper, we focus on diffusion model backbone which has been much neglected before. We systematically explore vision Transformers as diffusion learners for various generative tasks. With our improvements the performance of vanilla ViT-based backbone (IU-ViT) is boosted to be on par with traditional U-Net-based methods. We further provide a hypothesis on the implication of disentangling the generative backbone as an encoder-decoder structure and show proof-of-concept experiments verifying the effectiveness of a stronger encoder for generative tasks with ASymmetriC ENcoder Decoder (ASCEND). Our improvements achieve competitive results on CIFAR-10, CelebA, LSUN, CUB Bird and large-resolution text-to-image tasks. To the best of our knowledge, we are the first to successfully train a single diffusion model on text-to-image task beyond 64x64 resolution. We hope this will motivate people to rethink the modeling choices and the training pipelines for diffusion-based generative models.

📄 PDF Abstract BibTeX arXiv:2212.13771

Code (0)

등록된 구현이 없습니다.

Tasks

Decoder

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Denoising Diffusion Autoencoders are Unified Self-supervised Learners

2023-03-17 · ICCV 2023 1 · Weilai Xiang, Hongyu Yang, Di Huang, Yunhong Wang

Inspired by recent advances in diffusion models, which are reminiscent of denoising autoencoders, we investigate whether they can acquire discriminative representations for classification via generative pre-training. Thi…

Contrastive Learning Denoising Image Generation Linear evaluation+2

SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers

2024-01-16 · Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi 외

We present Scalable Interpolant Transformers (SiT), a family of generative models built on the backbone of Diffusion Transformers (DiT). The interpolant framework, which allows for connecting two distributions in a more …

Image Generation

An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

2024-06-13 · Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain, Martin R. Oswald 외

This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias -- locality in modern computer vision architectures. Concretely, we find that vanil…

Image Generation Inductive Bias Self-Supervised Learning

Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models

2022-06-07 · Walter H. L. Pinaya, Mark S. Graham, Robert Gray, Pedro F Da Costa 외

Deep generative models have emerged as promising tools for detecting arbitrary anomalies in data, dispensing with the necessity for manual labelling. Recently, autoregressive transformers have achieved state-of-the-art p…

Anomaly Detection Denoising

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

2024-04-02 · CVPR 2024 1 · Keon-Hee Park, Kyungwoo Song, Gyeong-Moon Park

Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challen…

class-incremental learning Class Incremental Learning Few-Shot Class-Incremental Learning Incremental Learning+2