Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction
Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall's rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.
Code (0)
등록된 구현이 없습니다.
Tasks
Grammatical Error CorrectionMachine TranslationText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
VAIYAKARANA : A Benchmark for Automatic Grammar Correction in Bangla
Bangla (Bengali) is the fifth most spoken language globally and, yet, the problem of automatic grammar correction in Bangla is still in its nascent stage. This is mostly due to the need for a large corpus of grammaticall…
SentenceRethinking the Roles of Large Language Models in Chinese Grammatical Error Correction
Recently, Large Language Models (LLMs) have been widely studied by researchers for their roles in various downstream NLP tasks. As a fundamental task in the NLP field, Chinese Grammatical Error Correction (CGEC) aims to …
Grammatical Error CorrectionRecording for Eyes, Not Echoing to Ears: Contextualized Spoken-to-Written Conversion of ASR Transcripts
Automatic Speech Recognition (ASR) transcripts exhibit recognition errors and various spoken language phenomena such as disfluencies, ungrammatical sentences, and incomplete sentences, hence suffering from poor readabili…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)In-Context Learningspeech-recognition+1ChatLang-8: An LLM-Based Synthetic Data Generation Framework for Grammatical Error Correction
We explore and improve the capabilities of LLMs to generate data for grammatical error correction (GEC). When merely producing parallel sentences, their patterns are too simplistic to be valuable as a corpus. To address …
Grammatical Error CorrectionSynthetic Data GenerationGrammatical Error Generation Based on Translated Fragments
We perform neural machine translation of sentence fragments in order to create large amounts of training data for English grammatical error correction. Our method aims at simulating mistakes made by second language learn…
Grammatical Error CorrectionMachine TranslationSentenceTranslation