Learning to Score System Summaries for Better Content Selection Evaluation.
The evaluation of summaries is a challenging but crucial task of the summarization field. In this work, we propose to learn an automatic scoring metric based on the human judgements available as part of classical summarization datasets like TAC-2008 and TAC-2009. Any existing automatic scoring metrics can be included as features, the model learns the combination exhibiting the best correlation with human judgments. The reliability of the new metric is tested in a further manual evaluation where we ask humans to evaluate summaries covering the whole scoring spectrum of the metric. We release the trained metric as an open-source tool.
Code (0)
등록된 구현이 없습니다.
Tasks
Document SummarizationMulti-Document SummarizationSemantic Textual SimilaritySimilar Papers 제목 키워드 기반
PSentScore: Evaluating Sentiment Polarity in Dialogue Summarization
Automatic dialogue summarization is a well-established task with the goal of distilling the most crucial information from human conversations into concise textual summaries. However, most existing research has predominan…
What comes next? Extractive summarization by next-sentence prediction
Existing approaches to automatic summarization assume that a length limit for the summary is given, and view content selection as an optimization problem to maximize informativeness and minimize redundancy within this bu…
Extractive SummarizationInformativenessPredictionSentenceUnderstanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries
Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary’s information quality by calculati…
Question AnsweringFrom Thumbnails to Summaries - A single Deep Neural Network to Rule Them All
Video summaries come in many forms, from traditional single-image thumbnails, animated thumbnails, storyboards, to trailer-like video summaries. Content creators use the summaries to display the most attractive portion o…
AllDecoderManagementJoint Optimization of User-desired Content in Multi-document Summaries by Learning from User Feedback
In this paper, we propose an extractive multi-document summarization (MDS) system using joint optimization and active learning for content selection grounded in user feedback. Our method interactively obtains user feedba…
Active LearningDocument SummarizationMulti-Document Summarization