Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics
Despite extensive recent advances in summary generation models, evaluation of auto-generated summaries still widely relies on single-score systems insufficient for transparent assessment and in-depth qualitative analysis. Towards bridging this gap, we propose the multifaceted interpretable summary evaluation method (MISEM), which is based on allocation of a summary's contextual token embeddings to semantic topics identified in the reference text. We further contribute an interpretability toolbox for automated summary evaluation and interactive visual analysis of summary scoring, topic identification, and token-topic allocation. MISEM achieves a promising .404 Pearson correlation with human judgment on the TAC'08 dataset.
Code (1)
Similar Papers 제목 키워드 기반
AMR4NLI: Interpretable and robust NLI measures from semantic graphs
The task of natural language inference (NLI) asks whether a given premise (expressed in NL) entails a given NL hypothesis. NLI benchmarks contain human ratings of entailment, but the meaning relationships driving these r…
Natural Language InferenceSentenceSUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization
We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). We propose SUPERT, which rates the q…
Document SummarizationMulti-Document Summarizationreinforcement-learningReinforcement Learning (RL)+2SLICE: Supersense-based Lightweight Interpretable Contextual Embeddings
Contextualised embeddings such as BERT have become de facto state-of-the-art references in many NLP applications, thanks to their impressive performances. However, their opaqueness makes it hard to interpret their behavi…
Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is designed to find and count all possible minute inconsistencies of the summary with respect to the sour…
PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure
In this work, we propose a novel tree-based explanation technique, PEACH (Pretrained-embedding Explanation Across Contextual and Hierarchical Structure), that can explain how text-based documents are classified by using …
Attributefeature selectiontext-classificationText Classification