Best practices for the human evaluation of automatically generated text
Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated. While there is some agreement regarding automatic metrics, there is a high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.
Code (0)
등록된 구현이 없습니다.
Tasks
Text GenerationSimilar Papers 제목 키워드 기반
AI-Assisted Assessment of Coding Practices in Modern Code Review
Modern code review is a process in which an incremental code contribution made by a code author is reviewed by one or more peers before it is committed to the version control system. An important element of modern code r…
Language ModelingLanguage ModellingLarge Language ModelLongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization
While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form su…
FormLeveraging Large Language Models for Zero-shot Lay Summarisation in Biomedicine and Beyond
In this work, we explore the application of Large Language Models to zero-shot Lay Summarisation. We propose a novel two-stage framework for Lay Summarisation based on real-life processes, and find that summaries generat…
ArticlesTo Err is AI : A Case Study Informing LLM Flaw Reporting Practices
In August of 2024, 495 hackers generated evaluations in an open-ended bug bounty targeting the Open Language Model (OLMo) from The Allen Institute for AI. A vendor panel staffed by representatives of OLMo's safety progra…
Language ModelingLanguage ModellingMetaLint: Easy-to-Hard Generalization for Code Linting
Large language models excel at code generation but struggle with code linting, particularly in generalizing to unseen or evolving best practices beyond those observed during training. We introduce MetaLint, a meta-learni…
Code Generation