Reference-less Quality Estimation of Text Simplification Systems
The evaluation of text simplification (TS) systems remains an open challenge. As the task has common points with machine translation (MT), TS is often evaluated using MT metrics such as BLEU. However, such metrics require high quality reference data, which is rarely available for TS. TS has the advantage over MT of being a monolingual task, which allows for direct comparisons to be made between the simplified text and its original version. In this paper, we compare multiple approaches to reference-less quality estimation of sentence-level text simplification systems, based on the dataset used for the QATS 2016 shared task. We distinguish three different dimensions: gram-maticality, meaning preservation and simplicity. We show that n-gram-based MT metrics such as BLEU and METEOR correlate the most with human judgment of grammaticality and meaning preservation, whereas simplicity is best evaluated by basic length-based metrics.
Code (1)
Tasks
Machine TranslationSentenceText SimplificationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Simple-QE: Better Automatic Quality Estimation for Text Simplification
Text simplification systems generate versions of texts that are easier to understand for a broader audience. The quality of simplified texts is generally estimated using metrics that compare to human references, which ca…
Text SimplificationSemantic Structural Evaluation for Text Simplification
Current measures for evaluating text simplification systems focus on evaluating lexical text aspects, neglecting its structural aspects. In this paper we propose the first measure to address structural aspects of text si…
Semantic ParsingSentenceText SimplificationvalidMaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplification
In text summarization and simplification, system outputs must be evaluated along multiple dimensions such as relevance, factual consistency, fluency, and grammaticality, and a wide range of possible outputs could be of h…
Language ModelingLanguage ModellingMasked Language ModelingText Simplification+1Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation
Text simplification intends to make a text easier to read while preserving its core meaning. Intuitively and as shown in previous works, these two dimensions (simplification and meaning preservation) are often-times inve…
SentenceText SimplificationREFeREE: A REference-FREE Model-Based Metric for Text Simplification
Text simplification lacks a universal standard of quality, and annotated reference simplifications are scarce and costly. We propose to alleviate such limitations by introducing REFeREE, a reference-free model-based metr…
Text Simplification