A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation
Automatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text qualityis represented in a variety of dimensions ranging from grammaticality to readability and coherence. In our work, we analyze the depen-dencies between a variety of quality dimensions on automatically created multi-document summaries and which dimensions automaticevaluation metrics such as ROUGE, PEAK or JSD are able to capture. Our results indicate that variants of ROUGE are correlated tovarious quality dimensions and that some automatic summarization methods achieve higher quality summaries than others with respectto individual summary quality dimensions. Our results also indicate that differentiating between quality dimensions facilitates inspectionand fine-grained comparison of summarization methods and its characteristics. We make the data from our two summarization qualityevaluation experiments publicly available in order to facilitate the future development of specialized automatic evaluation methods.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification
In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of h…
Language ModelingLanguage ModellingMasked Language ModelingText Simplification+1Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning
The evaluation of summary quality encompasses diverse dimensions such as consistency, coherence, relevance, and fluency. However, existing summarization methods often target a specific dimension, facing challenges in gen…
Multi-Objective Reinforcement LearningMulti-Task Learningreinforcement-learningReinforcement Learning+1Truth or Error? Towards systematic analysis of factual errors in abstractive summaries
This paper presents a typology of errors produced by automatic summarization systems. The typology was created by manually analyzing the output of four recent neural summarization systems. Our work is motivated by the gr…
AugAbEx: Bridging Abstractive and Extractive Legal Summarization
Automatic summarization of legal judgments liberates law professionals from heavy cognitive burden due to the complexity of the language, context-sensitive legal jargon, and the length of the document. Caveats of abstrac…
TL;DR Progress: Multi-faceted Literature Exploration in Text Summarization
This paper presents TL;DR Progress, a new tool for exploring the literature on neural text summarization. It organizes 514~papers based on a comprehensive annotation scheme for text summarization approaches and enables f…
Text Summarization