ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
Evaluation of summarization tasks is extremely crucial to determining the quality of machine generated summaries. Over the last decade, ROUGE has become the standard automatic evaluation measure for evaluating summarization tasks. While ROUGE has been shown to be effective in capturing n-gram overlap between system and human composed summaries, there are several limitations with the existing ROUGE measures in terms of capturing synonymous concepts and coverage of topics. Thus, often times ROUGE scores do not reflect the true quality of summaries and prevents multi-faceted evaluation of summaries (i.e. by topics, by overall content coverage and etc). In this paper, we introduce ROUGE 2.0, which has several updated measures of ROUGE: ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq and ROUGE-TopicUniq+Synonyms; all of which are improvements over the core ROUGE measures.
Code (3)
Similar Papers 제목 키워드 기반
HOLMS: Alternative Summary Evaluation with Large Language Models
Efficient document summarization requires evaluation measures that can not only rank a set of systems based on an average score, but also highlight which individual summary is better than another. However, despite the ve…
Document SummarizationExtractive SummarizationThe Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization
ROUGE is widely used to automatically evaluate summarization systems. However, ROUGE measures semantic overlap between a system summary and a human reference on word-string level, much at odds with the contemporary treat…
Document SummarizationNews SummarizationWord EmbeddingsAnswers Unite! Unsupervised Metrics for Reinforced Summarization Models
Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics th…
Abstractive Text SummarizationQuestion Answeringreinforcement-learningReinforcement Learning+1Revisiting Summarization Evaluation for Scientific Articles
Evaluation of text summarization approaches have been mostly based on metrics that measure similarities of system generated summaries with a set of human written gold-standard summaries. The most widely used metric in su…
ArticlesText SummarizationPerSum: Novel Systems for Document Summarization in Persian
In this paper we explore the problem of document summarization in Persian language from two distinct angles. In our first approach, we modify a popular and widely cited Persian document summarization framework to see how…
ArticlesDocument Summarization