Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning

2024-10-18 · Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, Lingpeng Kong

Autoregressive language models, despite their impressive capabilities, struggle with complex reasoning and long-term planning tasks. We introduce discrete diffusion models as a novel solution to these challenges. Through the lens of subgoal imbalance, we demonstrate how diffusion models effectively learn difficult subgoals that elude autoregressive approaches. We propose Multi-granularity Diffusion Modeling (MDM), which prioritizes subgoals based on difficulty during learning. On complex tasks like Countdown, Sudoku, and Boolean Satisfiability Problems, MDM significantly outperforms autoregressive models without using search techniques. For instance, MDM achieves 91.5\% and 100\% accuracy on Countdown and Sudoku, respectively, compared to 45.8\% and 20.7\% for autoregressive models. Our work highlights the potential of diffusion-based approaches in advancing AI capabilities for sophisticated language understanding and problem-solving tasks.

📄 PDF Abstract BibTeX arXiv:2410.14157

Code (1)

HKUNLP/diffusion-vs-ar 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

2025-10-07 · Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang 외 arxiv

We propose SDAR, a Synergistic Diffusion-Autoregression paradigm that unifies the training efficiency of autoregressive models with the parallel inference capability of diffusion. Instead of costly end-to-end diffusion t…

MonoFormer: One Transformer for Both Diffusion and Autoregression

2024-09-24 · Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng 외

Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use a…

Image Generation Text Generation

On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond

2025-10-07 · Chenxiao Yang, Cai Zhou, David Wipf, Zhiyuan Li arxiv

Diffusion language models have recently emerged as a competitive alternative to autoregressive language models. Beyond next-token generation, they are more efficient and flexible by enabling parallel and any-order token …

Discrete Diffusion Language Models for Interactive Radiology Report Drafting

2026-07-01 · Max Van Puyvelde, Halil Ibrahim Gulluk, Wim Van Criekinge, Olivier Gevaert arxiv

Diffusion language models, which generate text by denoising a token canvas bidirectionally instead of emitting tokens left to right, have become competitive with autoregressive (AR) generation. Medical foundation models,…

Visual Question Answering

GryphOne: Symbol-Aware Masked Diffusion for Structural Refinement in Offline Handwritten Mathematical Expression Recognition

2026-02-03 · Takaya Kawakatsu, Ryo Ishiyama arxiv

Handwritten mathematical expression recognition (HMER) requires reasoning over diverse symbols and structures, yet autoregressive models struggle with exposure bias and syntax inconsistency. We present GryphOne, a discre…