Flesch-Kincaid is Not a Text Simplification Evaluation Metric
Sentence-level text simplification is currently evaluated using both automated metrics and human evaluation. For automatic evaluation, a combination of metrics is usually employed to evaluate different aspects of the simplification. Flesch-Kincaid Grade Level (FKGL) is one metric that has been regularly used to measure the readability of system output. In this paper, we argue that FKGL should not be used to evaluate text simplification systems. We provide experimental analyses on recent system output showing that the FKGL score can easily be manipulated to improve the score dramatically with only minor impact on other automated metrics (BLEU and SARI). Instead of using FKGL, we suggest that the component statistics, along with others, be used for posthoc analysis to understand system behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceText SimplificationSimilar Papers 제목 키워드 기반
Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts
Using large language models (LLMs) for educational applications like dialogue-based teaching is a hot topic. Effective teaching, however, requires teachers to adapt the difficulty of content and explanations to the educa…
The Common Readability Formula & Five Adjusted Readability Formulas for Text Simplification, Medical Documents and Other General Uses
Traditional readability formulas, or equations, are inaccurate and measure highly limited linguistic properties. Despite the recent machine learning-based readability assessment models, many researchers insist on using t…
Text SimplificationTraditional Readability Formulas Compared for English
Traditional English readability formulas, or equations, were largely developed in the 20th century. Nonetheless, many researchers still rely on them for various NLP applications. This phenomenon is presumably due to the …
NeRFText SimplificationFlesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models
Flesch Reading Ease (FRE) and the Flesch-Kincaid Grade Level (FKGL) are widely used readability scores for English computed from the same two document statistics, yet their stability on long documents need not imply inva…
Topic ModelsFlesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models
Readability metrics and standards such as Flesch Kincaid Grade Level (FKGL) and the Common European Framework of Reference for Languages (CEFR) exist to guide teachers and educators to properly assess the complexity of e…