SOME: Reference-less Sub-Metrics Optimized for Manual Evaluations of Grammatical Error Correction
We propose a reference-less metric trained on manual evaluations of system outputs for grammatical error correction (GEC). Previous studies have shown that reference-less metrics are promising; however, existing metrics are not optimized for manual evaluations of the system outputs because no dataset of the system output exists with manual evaluation. This study manually evaluates outputs of GEC systems to optimize the metrics. Experimental results show that the proposed metric improves correlation with the manual evaluation in both system- and sentence-level meta-evaluation. Our dataset and metric will be made publicly available.
Code (1)
Tasks
Grammatical Error CorrectionSentenceSimilar Papers 제목 키워드 기반
Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems
In grammatical error correction (GEC), automatically evaluating system outputs requires gold-standard references, which must be created manually and thus tend to be both expensive and limited in coverage. To address this…
Grammatical Error CorrectionMachine TranslationAutomatic Reference-Based Evaluation of Pronoun Translation Misses the Point
We compare the performance of the APT and AutoPRF metrics for pronoun translation against a manually annotated dataset comprising human judgements as to the correctness of translations of the PROTEST test suite. Although…
TranslationResults of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges
This paper presents the results of the WMT19 Metrics Shared Task. Participants were asked to score the outputs of the translations systems competing in the WMT19 News Translation Task with automatic metrics. 13 research …
TranslationFaceDig: Automated tool for placing landmarks on facial portraits for geometric morphometrics users
Landmark digitization is essential in geometric morphometrics, enabling the quantification of biological shapes, such as facial structures, for in-depth morphological analysis. Traditional landmarking, which identifies s…
Morphological AnalysisLEPOR: An Augmented Machine Translation Evaluation Metric
Machine translation (MT) was developed as one of the hottest research topics in the natural language processing (NLP) literature. One important issue in MT is that how to evaluate the MT system reasonably and tell us whe…
Machine TranslationPOSTranslation