Unlocking Prompt Infilling Capability for Diffusion Language Models
Masked diffusion language models (dLMs) generate text through bidirectional denoising, yet this capability remains locked for infilling prompts. This limitation is an artifact of the current supervised finetuning (SFT) convention of applying response-only masking. To unlock this capability, we extend full-sequence masking during SFT, where both prompts and responses are masked jointly. Once unlocked, the model infills masked portions of a prompt template conditioned on few-shot examples. We show that such model-infilled prompts match or surpass manually designed templates, transfer effectively across models, and are complementary to existing prompt optimization methods. Our results suggest that training practices, not architectural limitations, are the primary bottleneck preventing masked diffusion language models from infilling effective prompts
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Unlocking the Potential of Diffusion Language Models through Template Infilling
Diffusion Language Models (DLMs) have emerged as a promising alternative to Autoregressive Language Models, yet their inference strategies remain limited to prefix-based prompting inherited from the autoregressive paradi…
Mathematical ReasoningCode GenerationFlexible-length Text Infilling for Discrete Diffusion Models
Discrete diffusion models are a new class of text generators that offer advantages such as bidirectional context use, parallelizable generation, and flexible prompting compared to autoregressive models. However, a critic…
PositionText InfillingDreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a cr…
EFIM: Efficient Serving of LLMs for Infilling Tasks with Improved KV Cache Reuse
Large language models (LLMs) are often used for infilling tasks, which involve predicting or generating missing information in a given text. These tasks typically require multiple interactions with similar context. To re…
Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly
Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling, yet their performance is constrained by the pre-specified infilling length. In this paper, we reveal that DLMs…