Inherent Biases in Reference based Evaluation for Grammatical Error Correction and Text Simplification
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality ({\it low coverage bias} or LCB). This paper shows that overcoming LCB in Grammatical Error Correction (GEC) evaluation cannot be attained by re-scaling or by increasing the number of references in any feasible range, contrary to previous suggestions. This is due to the long-tailed distribution of valid corrections for a sentence. Concretely, we show that LCB incentivizes GEC systems to avoid correcting even when they can generate a valid correction. Consequently, existing systems obtain comparable or superior performance compared to humans, by making few but targeted changes to the input. Similar effects on Text Simplification further support our claims.
Code (1)
Tasks
Grammatical Error CorrectionSentenceText GenerationText SimplificationvalidSimilar Papers 제목 키워드 기반
Inherent Biases in Reference-based Evaluation for Grammatical Error Correction
The prevalent use of too few references for evaluating text-to-text generation is known to bias estimates of their quality (henceforth, low coverage bias or LCB). This paper shows that overcoming LCB in Grammatical Error…
Grammatical Error CorrectionSentenceText GenerationText Simplification+1Evaluation of really good grammatical error correction
Although rarely stated, in practice, Grammatical Error Correction (GEC) encompasses various models with distinct objectives, ranging from grammatical error detection to improving fluency. Traditional evaluation methods f…
Grammatical Error CorrectionGrammatical Error DetectionIMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator
We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing a…
Grammatical Error CorrectionGrammatical Error DetectionLanguage ModelingLanguage Modelling+1There's No Comparison: Reference-less Evaluation Metrics in Grammatical Error Correction
Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the go…
BenchmarkingGrammatical Error CorrectionSentenceIs this the end of the gold standard? A straightforward reference-less grammatical error correction metric
It is difficult to rank and evaluate the performance of grammatical error correction (GEC) systems, as a sentence can be rewritten in numerous correct ways. A number of GEC metrics have been used to evaluate proposed GEC…
Grammatical Error CorrectionSentence