Distillation of Discrete Diffusion through Dimensional Correlations

2024-10-11 · Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, Yuki Mitsufuji

Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature. While this issue is being addressed in continuous domains, discrete diffusion models face unique challenges, particularly in capturing dependencies between elements (e.g., pixel relationships in image, sequential dependencies in language) mainly due to the computational cost of processing high-dimensional joint distributions. In this paper, (i) we propose "mixture" models for discrete diffusion that are capable of treating dimensional correlations while remaining scalable, and (ii) we provide a set of loss functions for distilling the iterations of existing models. Two primary theoretical insights underpin our approach: First, conventional models with element-wise independence can well approximate the data distribution, but essentially require {\it many sampling steps}. Second, our loss functions enable the mixture models to distill such many-step conventional models into just a few steps by learning the dimensional correlations. Our experimental results show the effectiveness of the proposed method in distilling pretrained discrete diffusion models across image and language domains. The code used in the paper is available at https://github.com/sony/di4c .

📄 PDF Abstract BibTeX arXiv:2410.08709

Code (1)

sony/di4c 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

2026-05-22 · Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis, Morteza Mardani 외 arxiv

Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant o…

Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens

2026-03-19 · Yuqing Wang, Chuofan Ma, Zhijie Lin, Yao Teng 외 arxiv

Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discret…

Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD

2026-03-20 · Emiel Hoogeboom, David Ruhe, Jonathan Heek, Thomas Mensink 외 arxiv

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Mome…

Discrete Diffusion with Sample-Efficient Estimators for Conditionals

2026-02-23 · Karthik Elamvazhuthi, Abhijith Jayakumar, Andrey Y. Lokhov arxiv

We study a discrete denoising diffusion framework that integrates a sample-efficient estimator of single-site conditionals with round-robin noising and denoising dynamics for generative modeling over discrete state space…

Density Estimation

One-Step Distillation of Discrete Diffusion Image Generators via Fixed-Point Iteration

2026-05-20 · Chaoyang Wang, Yunhai Tong arxiv

Discrete diffusion models excel at visual synthesis but rely on slow, iterative decoding. Existing single-step distillation methods attempt to bypass this bottleneck, either by training auxiliary score networks that effe…