FEANEL: A Benchmark for Fine-Grained Error Analysis in K-12 English Writing
Large Language Models (LLMs) have transformed artificial intelligence, offering profound opportunities for educational applications. However, their ability to provide fine-grained educational feedback for K-12 English writing remains underexplored. In this paper, we challenge the error analysis and pedagogical skills of LLMs by introducing the problem of Fine-grained Error Analysis for English Learners and present the Fine-grained Error ANalysis for English Learners (FEANEL) Benchmark. The benchmark comprises 1,000 essays written by elementary and secondary school students, and a well-developed English writing error taxonomy. Each error is annotated by language education experts and categorized by type, severity, and explanatory feedback, using a part-of-speech-based taxonomy they co-developed. We evaluate state-of-the-art LLMs on the FEANEL Benchmark to explore their error analysis and pedagogical abilities. Experimental results reveal significant gaps in current LLMs' ability to perform fine-grained error analysis, highlighting the need for advancements in particular methods for educational applications.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 English
Modern named entity recognition systems have steadily improved performance in the age of larger and more powerful neural models. However, over the past several years, the state-of-the-art has seemingly hit another platea…
named-entity-recognitionNamed Entity RecognitionNERELEVANT: A Fully Automatic Fine-Grained Entity Linking Evaluation and Analysis Tool
We present Elevant, a tool for the fully automatic fine-grained evaluation of a set of entity linkers on a set of benchmarks. Elevant provides an automatic breakdown of the performance by various error categories and by …
Entity LinkingDetecting Pipeline Failures through Fine-Grained Analysis of Web Agents
Web agents powered by large language models (LLMs) can autonomously perform complex, multistep tasks in dynamic web environments. However, current evaluations mostly focus on the overall success while overlooking interme…
Fine-grained Hallucination Detection and Editing for Language Models
Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. In this paper, we introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in di…
HallucinationRetrievalCOHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts
In recent years, Multimodal Large Language Models (MLLMs) have achieved remarkable progress on a wide range of multimodal benchmarks. Despite these advances, most existing benchmarks mainly focus on single-image or multi…