Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation
We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design Quintd - a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.
Code (0)
등록된 구현이 없습니다.
Tasks
Data-to-Text GenerationIn-Context LearningText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
Recent technological advances in brain recording and artificial intelligence are propelling a new paradigm in neuroscience beyond the traditional controlled experiment. Rather than focusing on cued, repeated trials, natu…
Experimental DesignOn the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe
Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-…
In-Context LearningReverse DictionarySimulating and analyzing a sparse order book: an application to intraday electricity markets
This paper presents a novel model for simulating and analyzing sparse limit order books (LOBs), with a specific application to the European intraday electricity market. In illiquid markets, characterized by significant g…
Beyond Black-Box Benchmarking: Observability, Analytics, and Optimization of Agentic Systems
The rise of agentic AI systems, where agents collaborate to perform diverse tasks, poses new challenges with observing, analyzing and optimizing their behavior. Traditional evaluation and benchmarking approaches struggle…
BenchmarkingPosition: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamics
What would it mean to have a scientific understanding of AI? Models are not static objects: they are snapshots of time-evolving processes shaped by data, objectives, architectures, and optimization dynamics. Yet much of …