Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect. First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries annotated by humans, which is currently standard practice. We demonstrate how this small change leads to more precise estimates of system-level correlations. Second, we propose to calculate correlations only on pairs of systems which are separated by differences in automatic scores that are commonly used to argue one system is of higher quality. This allows us to demonstrate that our best estimate of the correlation of ROUGE to human judgments is near 0 in realistic scenarios. Finally, the results from both analyses point to the need for future research to focus on developing more consistent and reliable human evaluations of summaries.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correla…
Video Joint Modelling Based on Hierarchical Transformer for Co-summarization
Video summarization aims to automatically generate a summary (storyboard or video skim) of a video, which can facilitate large-scale video retrieval and browsing. Most of the existing methods perform video summarization …
RetrievalSupervised Video SummarizationVideo RetrievalVideo Summarization+1Examining the State-of-the-Art in News Timeline Summarization
Previous work on automatic news timeline summarization (TLS) leaves an unclear picture about how this task can generally be approached and how well it is currently solved. This is mostly due to the focus on individual su…
Timeline SummarizationWhat's under the hood: Investigating Automatic Metrics on Meeting Summarization
Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific erro…
Meeting SummarizationUnsupervised Abstractive Dialogue Summarization for Tete-a-Tetes
High-quality dialogue-summary paired data is expensive to produce and domain-sensitive, making abstractive dialogue summarization a challenging task. In this work, we propose the first unsupervised abstractive dialogue s…
Abstractive Dialogue Summarizationdialogue summarySentenceText Summarization