An Automated Length-Aware Quality Metric for Summarization
This paper proposes NOrmed Index of Retention (NOIR), a quantitative objective metric for evaluating summarization quality of arbitrary texts that relies on both the retention of semantic meaning and the summary length compression. This gives a measure of how well the recall-compression tradeoff is managed, the most important skill in summarization. Experiments demonstrate that NOIR effectively captures the token-length / semantic retention tradeoff of a summarizer and correlates to human perception of sumarization quality. Using a language model-embedding to measure semantic similarity, it provides an automated alternative for assessing summarization quality without relying on time-consuming human-generated reference summaries. The proposed metric can be applied to various summarization tasks, offering an automated tool for evaluating and improving summarization algorithms, summarization prompts, and synthetically-generated summaries.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic SimilaritySimilar Papers 제목 키워드 기반
Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment
Automated essay scoring (AES) enables scalable assessment and timely feedback but remains challenged by transformer input-length limitations, which can cause information loss when processing long essays. This study propo…
Automated Essay ScoringSemantic SimilarityLegal Extractive Summarization of U.S. Court Opinions
This paper tackles the task of legal extractive summarization using a dataset of 430K U.S. court opinions with key passages annotated. According to automated summary quality metrics, the reinforcement-learning-based MemS…
Extractive Summarizationreinforcement-learningAutomated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations
Evaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across differ…
Document SummarizationMulti-Document SummarizationQuality of syntactic implication of RL-based sentence summarization
Work on summarization has explored both reinforcement learning (RL) optimization using ROUGE as a reward and syntax-aware models, such as models those input is enriched with part-of-speech (POS)-tags and dependency infor…
POSReinforcement LearningReinforcement Learning (RL)Sentence+1Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive Summarization
The ROUGE metric is commonly used to evaluate extractive summarization task, but it has been criticized for its lack of semantic awareness and its ignorance about the ranking quality of the extractive summarizer. Previou…
DiversityExtractive Summarization