The Feasibility of Embedding Based Automatic Evaluation for Single Document Summarization
ROUGE is widely used to automatically evaluate summarization systems. However, ROUGE measures semantic overlap between a system summary and a human reference on word-string level, much at odds with the contemporary treatment of semantic meaning. Here we present a suite of experiments on using distributed representations for evaluating summarizers, both in reference-based and in reference-free setting. Our experimental results show that the max value over each dimension of the summary ELMo word embeddings is a good representation that results in high correlation with human ratings. Averaging the cosine similarity of all encoders we tested yields high correlation with manual scores in reference-free setting. The distributed representations outperform ROUGE in recent corpora for abstractive news summarization but are less good on test data used in past evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
Document SummarizationNews SummarizationWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GLEAKE: Global and Local Embedding Automatic Keyphrase Extraction
Automated methods for granular categorization of large corpora of text documents have become increasingly more important with the rate scientific, news, medical, and web documents are growing in the last few years. Autom…
Document SummarizationInformation RetrievalKeyphrase ExtractionRetrievalReconstructing Manual Information Extraction with DB-to-Document Backprojection: Experiments in the Life Science Domain
We introduce a novel scientific document processing task for making previously inaccessible information in printed paper documents available to automatic processing. We describe our data set of scanned documents and data…
Automatic Inter-document Multi-hop Scientific QA Generation
Existing automatic scientific question generation studies mainly focus on single-document factoid QA, overlooking the inter-document reasoning crucial for scientific understanding. We present AIM-SciQA, an automated fram…
Machine Reading ComprehensionQuestion GenerationDetecting (Un)Important Content for Single-Document News Summarization
We present a robust approach for detecting intrinsic sentence importance in news, by training on two corpora of document-summary pairs. When used for single-document summarization, our approach, combined with the "beginn…
Document SummarizationNews SummarizationSentenceSingle-Document Summarization Using Sentence Embeddings and K-Means Clustering
This paper proposes a novel method for extractive single document summarization using K-Means clustering and Sentence Embeddings. Sentence embeddings were processed by K-Means algorithm into a number of clusters dependin…
ClusteringDocument SummarizationregressionSentence+1