CLEME2.0: Towards More Interpretable Evaluation by Disentangling Edits for Grammatical Error Correction
The paper focuses on improving the interpretability of Grammatical Error Correction (GEC) metrics, which receives little attention in previous studies. To bridge the gap, we propose CLEME2.0, a reference-based evaluation strategy that can describe four elementary dimensions of GEC systems, namely hit-correction, error-correction, under-correction, and over-correction. They collectively contribute to revealing the critical characteristics and locating drawbacks of GEC systems. Evaluating systems by Combining these dimensions leads to high human consistency over other reference-based and reference-less metrics. Extensive experiments on 2 human judgement datasets and 6 reference datasets demonstrate the effectiveness and robustness of our method. All the codes will be released after the peer review.
Code (1)
Tasks
Grammatical Error CorrectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction
Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity. Designing an evaluation metric that is as objective as possible is crucial to the development of GEC…
Grammatical Error CorrectionDisentangling Content and Motion for Text-Based Neural Video Manipulation
Giving machines the ability to imagine possible new objects or scenes from linguistic descriptions and produce their realistic renderings is arguably one of the most challenging problems in computer vision. Recent advanc…
Encirclement Guaranteed Finite-Time Capture against Unknown Evader Strategies
We consider a pursuit-evasion scenario involving a group of pursuers and a single evader in a two-dimensional unbounded environment. The pursuers aim to capture the evader in finite time while ensuring the evader remains…
Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation
Automatic evaluation in grammatical error correction (GEC) is crucial for selecting the best-performing systems. Currently, reference-based metrics are a popular choice, which basically measure the similarity between hyp…
Grammatical Error CorrectionTERL: Large-Scale Multi-Target Encirclement Using Transformer-Enhanced Reinforcement Learning
Pursuit-evasion (PE) problem is a critical challenge in multi-robot systems (MRS). While reinforcement learning (RL) has shown its promise in addressing PE tasks, research has primarily focused on single-target pursuit, …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)