Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction

2024-05-21 · Maciej Kilian, Varun Jampani, Luke Zettlemoyer

Nearly every recent image synthesis approach, including diffusion, masked-token prediction, and next-token prediction, uses a Transformer network architecture. Despite this common backbone, there has been no direct, compute controlled comparison of how these approaches affect performance and efficiency. We analyze the scalability of each approach through the lens of compute budget measured in FLOPs. We find that token prediction methods, led by next-token prediction, significantly outperform diffusion on prompt following. On image quality, while next-token prediction initially performs better, scaling trends suggest it is eventually matched by diffusion. We compare the inference compute efficiency of each approach and find that next token prediction is by far the most efficient. Based on our findings we recommend diffusion for applications targeting image quality and low latency; and next-token prediction when prompt following or throughput is more important.

📄 PDF Abstract BibTeX arXiv:2405.13218

Code (0)

등록된 구현이 없습니다.

Tasks

Image Generation Prediction

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Multi-Head Attention 설명 없음

Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis

2026-06-29 · Shufan Li, Greg Heinrich, Hanrong Ye, Yonggan Fu 외 arxiv

We propose Nemotron-Labs-Diffusion-Image, a state-of-the-art masked discrete diffusion model (MDM) for high-resolution text-to-image synthesis. Compared with prior work on masked image generation, Nemotron-Labs-Diffusion…

Image Generation

MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer

2023-03-25 · ICCV 2023 1 · ShangHua Gao, Pan Zhou, Ming-Ming Cheng, Shuicheng Yan

Despite its success in image synthesis, we observe that diffusion probabilistic models (DPMs) often lack contextual reasoning ability to learn the relations among object parts in an image, leading to a slow learning proc…

Image Generation

Structure-Guided Histopathology Synthesis via Dual-LoRA Diffusion

2026-03-04 · Xuan Xu, Prateek Prasanna arxiv

Histopathology image synthesis plays an important role in tissue restoration, data augmentation, and modeling of tumor microenvironments. However, existing generative methods typically address restoration and generation …

Data Augmentation

Pyramid Hierarchical Masked Diffusion Model for Imaging Synthesis

2025-07-22 · Xiaojiao Xiao, Qinmin Vivian Hu, Guanghui Wang arxiv

Medical image synthesis plays a crucial role in clinical workflows, addressing the common issue of missing imaging modalities due to factors such as extended scan times, scan corruption, artifacts, patient motion, and in…

DiffRF: Rendering-Guided 3D Radiance Field Diffusion

2022-12-02 · CVPR 2023 1 · Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò 외

We introduce DiffRF, a novel approach for 3D radiance field synthesis based on denoising diffusion probabilistic models. While existing diffusion-based methods operate on images, latent codes, or point cloud data, we are…

Denoising