Human vs Automatic Metrics: on the Importance of Correlation Design
This paper discusses two existing approaches to the correlation analysis between automatic evaluation metrics and human scores in the area of natural language generation. Our experiments show that depending on the usage of a system- or sentence-level correlation analysis, correlation results between automatic scores and human judgments are inconsistent.
Code (1)
Tasks
SentenceText GenerationSimilar Papers 제목 키워드 기반
LCEval: Learned Composite Metric for Caption Evaluation
Automatic evaluation metrics hold a fundamental importance in the development and fine-grained analysis of captioning systems. While current evaluation metrics tend to achieve an acceptable correlation with human judgeme…
SentenceA Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics
The explosion of open-sourced models and Question-Answering (QA) datasets emphasizes the importance of automated QA evaluation. We studied the statistics of the existing evaluation metrics for a better understanding of t…
Question AnsweringNLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist
In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are …
Controllable Language ModellingDialogue GenerationLanguage Modellingnlg evaluation+3Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge
The effectiveness of automatic evaluation of generative models is typically measured by comparing the labels generated via automation with labels by humans using correlation metrics. However, metrics like Krippendorff's …
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correla…