MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification
In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of high quality. These properties make the development of an adaptable, reference-less evaluation metric both necessary and challenging. We introduce MaskEval, a reference-less metric for text summarization and simplification that operates by performing masked language modeling (MLM) on the concatenation of the candidate and the source texts. It features an attention-like weighting mechanism to modulate the relative importance of each MLM step, which crucially allows it to be adapted to evaluate different quality dimensions. We demonstrate its effectiveness on English summarization and simplification in terms of correlations with human judgments, and explore transfer scenarios between the two tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMasked Language ModelingText SimplificationText SummarizationSimilar Papers 제목 키워드 기반
Can summarization approximate simplification? A gold standard comparison
This study explores the overlap between text summarization and simplification outputs. While summarization evaluation methods are streamlined, simplification lacks cohesion, prompting the question: how closely can abstra…
Abstractive Text SummarizationText SummarizationSATS: simplification aware text summarization of scientific documents
Simplifying summaries of scholarly publications has been a popular method for conveying scientific discoveries to a broader audience. While text summarization aims to shorten long documents, simplification seeks to reduc…
Text SimplificationText SummarizationExploiting Summarization Data to Help Text Simplification
One of the major problems with text simplification is the lack of high-quality data. The sources of simplification datasets are limited to Wikipedia and Newsela, restricting further development of this field. In this pap…
SentenceText SimplificationText SummarizationA Semantic Relevance Based Neural Network for Text Summarization and Text Simplification
Text summarization and text simplification are two major ways to simplify the text for poor readers, including children, non-native speakers, and the functionally illiterate. Text summarization is to produce a brief summ…
DecoderSemantic SimilaritySemantic Textual SimilarityText Generation+2Combining Paraphrase Pre-trained Model and Controllable Rules for Unsupervised Sentence Simplification
Although neural sequence-to-sequence models for sentence simplification achieve some progress, they still suffer from the data sparsity problem and are lack of controllability. This paper proposes a two-stage approach fo…
SentenceText SimplificationText Summarization