CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization
To broaden the dissemination of scientific knowledge to diverse audiences, scientific document summarization must simultaneously control multiple attributes such as length and empirical focus. However, existing research typically focuses on controlling single attributes, leaving the compositional control of multiple attributes underexplored. To address this gap, we introduce CCSBench, a benchmark for compositional controllable summarization in the scientific domain. Our benchmark enables fine-grained control over both explicit attributes (e.g., length), which are objective and straightforward, and implicit attributes (e.g., empirical focus), which are more subjective and conceptual. We conduct extensive experiments on GPT-4, LLaMA2, and other popular LLMs under various settings. Our findings reveal significant limitations in large language models' ability to balance trade-offs between control attributes, especially implicit ones that require deeper understanding and abstract reasoning.
Code (0)
등록된 구현이 없습니다.
Tasks
Document SummarizationScientific Document SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Compositional Text-to-Image Generation with Dense Blob Representations
Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denote…
Image GenerationIn-Context LearningText to Image GenerationText-to-Image GenerationLexInstructEval: Lexical Instruction Following Evaluation for Large Language Models
The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significa…
Instruction FollowingLCTG Bench: LLM Controlled Text Generation Benchmark
The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructio…
Text GenerationFunny or Persuasive, but Not Both: Evaluating Fine-Grained Multi-Concept Control in LLMs
Large Language Models (LLMs) offer strong generative capabilities, but many applications require explicit and \textit{fine-grained} control over specific textual concepts, such as humor, persuasiveness, or formality. Pri…
Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?
In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries su…
Language ModelingLanguage ModellingLarge Language ModelReview Generation