Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks

2022-10-27 · Edwin Zhang, Yujie Lu, Shinda Huang, William Wang, Amy Zhang

Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have allowed for improved scaling along one or two of these axes, but are still computationally prohibitive to use. In this paper, we propose to address all three axes by leveraging \textbf{L}anguage to \textbf{C}ontrol \textbf{D}iffusion models as a hierarchical planner conditioned on language (LCD). We effectively and efficiently scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long horizon control problems conditioned on natural language instructions, as a step towards generalist agents. Comparing LCD with other state-of-the-art models on the CALVIN language robotics benchmark finds that LCD outperforms other SOTA methods in multi-task success rates, whilst improving inference speed over other comparable diffusion models by 3.3x~15x. We show that LCD can successfully leverage the unique strength of diffusion models to produce coherent long range plans while addressing their weakness in generating low-level details and control.

📄 PDF Abstract BibTeX arXiv:2210.15629

Code (1)

ezhang7423/language-control-diffusion 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Inference-time Scaling of Diffusion Models through Classical Search

2025-05-29 · Xiangcheng Zhang, Haowei Lin, Haotian Ye, James Zou 외

Classical search algorithms have long underpinned modern artificial intelligence. In this work, we tackle the challenge of inference-time control in diffusion models -- adapting generated outputs to meet diverse test-tim…

Image Generation Navigate

Likelihood-Based Diffusion Language Models

2023-05-30 · NeurIPS 2023 11 · Ishaan Gulrajani, Tatsunori B. Hashimoto

Despite a growing interest in diffusion-based language models, existing work has not shown that these models can attain nontrivial likelihoods on standard language modeling benchmarks. In this work, we take the first ste…

Language Modeling Language Modelling

ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies

2026-06-30 · Andrew Zou Li, Gokul Swamy, Yonatan Bisk, Andrea Bajcsy arxiv

Generative control policies (GCPs), such as diffusion policies and flow-based vision-language-action models, enable test-time scaling in robot control. Test-time compute can be allocated along two axes: sequential scalin…

Reinforcement Learning Robot Manipulation

Scaling Properties of Text Conditioning in Visual Generation

2026-07-31 · Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan 외 hf

We study empirical scaling properties for text conditioning in visual generation. Such properties have rarely been measured because diffusion loss does not scale with the number of tokens in natural-language prompts. Sur…

The Efficiency Gap in Byte Modeling

2026-05-13 · Celine Lee, Jing Nathan Yan, Chen Liang, Jiaxin Shi 외 arxiv

Modern language models have historically relied on two dominant design choices: subword tokenization and autoregressive (AR) ordering. These design decisions bake in priors that dictate a model's learning. Recently, two …