Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Advancements in natural language generation (NLG) and large language models (LLMs) have led to proficient text generation in various tasks. However, integrating intricate constraints into neural text generation, due to LLMs' opacity, remains challenging. This study investigates constrained text generation for LLMs, where predefined constraints are applied during LLM's generation process. Our research mainly focuses on mainstream open-source LLMs, categorizing constraints into lexical, structural, and relation-based types. We also present various benchmarks to facilitate fair evaluation. The study addresses some key research questions, including evaluating, understanding and improving constrained text generation for LLMs. Results illuminate LLMs' capacity and deficiency to incorporate constraints and provide insights for future developments in constrained text generation. Codes and datasets will be released upon acceptance.
Code (0)
등록된 구현이 없습니다.
Tasks
Text GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling
Recent visual generation models have made major progress in photorealism, typography, instruction following, and interactive editing, yet they still struggle with spatial reasoning, persistent state, long-horizon consist…
Instruction FollowingSpatial ReasoningXL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies
Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accom…
Long-Context UnderstandingReading ComprehensionUHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation
Large language models (LLMs) have emerged as pivotal contributors in contemporary natural language processing and are increasingly being applied across a diverse range of industries. However, these large-scale probabilis…
BenchmarkingHallucinationHallucination EvaluationText GenerationSRS-Stories: Vocabulary-constrained multilingual story generation for language learning
In this paper, we use large language models to generate personalized stories for language learners, using only the vocabulary they know. The generated texts are specifically written to teach the user new vocabulary by si…
Story GenerationEvaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets
We present a comprehensive evaluation of structured decoding for text-to-table generation with large language models (LLMs). While previous work has primarily focused on unconstrained generation of tables, the impact of …