An Anchor-Based Automatic Evaluation Metric for Document Summarization
The widespread adoption of reference-based automatic evaluation metrics such as ROUGE has promoted the development of document summarization. In this paper, we consider a new protocol for designing reference-based metrics that require the endorsement of source document(s). Following protocol, we propose an anchored ROUGE metric fixing each summary particle on source document, which bases the computation on more solid ground. Empirical results on benchmark datasets validate that source document helps to induce a higher correlation with human judgments for ROUGE metric. Being self-explanatory and easy-to-implement, the protocol can naturally foster various effective designs of reference-based metrics besides the anchored ROUGE introduced here.
Code (0)
등록된 구현이 없습니다.
Tasks
Document SummarizationSimilar Papers 제목 키워드 기반
An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics
Long documents such as academic articles and business reports have been the standard format to detail out important issues and complicated subjects that require extra attention. An automatic summarization system that can…
ArticlesDocument SummarizationText SummarizationA Study on Summarizing and Evaluating Long Documents
Text summarization has been a key language generation task for over 60 years. The field has advanced considerably during the past two years, benefiting from the proliferation of pre-trained Language Models (LMs). However…
Document SummarizationText GenerationText SummarizationA Comparative Study of Quality Evaluation Methods for Text Summarization
Evaluating text summarization has been a challenging task in natural language processing (NLP). Automatic metrics which heavily rely on reference summaries are not suitable in many situations, while human evaluation is t…
Text SummarizationA Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation
Automatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text qualityis represented in a variety of dimensions ranging from grammaticality to readabili…
Improving Factual Consistency of Abstractive Summarization via Question Answering
A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may pro…
Abstractive Text SummarizationQuestion Answering