Improving LLM Classification of Logical Errors by Integrating Error Relationship into Prompts
LLMs trained in the understanding of programming syntax are now providing effective assistance to developers and are being used in programming education such as in generation of coding problem examples or providing code explanations. A key aspect of programming education is understanding and dealing with error message. However, 'logical errors' in which the program operates against the programmer's intentions do not receive error messages from the compiler. In this study, building on existing research on programming errors, we first define the types of logical errors that can occur in programming in general. Based on the definition, we propose an effective approach for detecting logical errors with LLMs that makes use of relations among error types in the Chain-of-Thought and Tree-of-Thought prompts. The experimental results indicate that when such logical error descriptions in the prompt are used, the average classifition performance is about 21% higher than the ones without them. We also conducted an experiment for exploiting the relations among errors in generating a new logical error dataset using LLMs. As there is very limited dataset for logical errors such benchmark dataset can be very useful for various programming related applications. We expect that our work can assist novice programmers in identifying the causes of code errors and correct them more effectively.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Exploring word embeddings and phonological similarity for the unsupervised correction of language learner errors
The presence of misspellings and other errors or non-standard word forms poses a considerable challenge for NLP systems. Although several supervised approaches have been proposed previously to normalize these, annotated …
Grapheme-to-Phoneme ConversionLanguage AcquisitionWord EmbeddingsCyclocopula Technique to Study the Relationship Between Two Cyclostationary Time Series with Fractional Brownian Motion Errors
Detection of the relationship between two time series is so important in environmental and hydrological studies. Several parametric and non-parametric approaches can be applied to detect relationships. These techniques a…
Time SeriesTime Series AnalysisFrom Implicit to Explicit: Token-Efficient Logical Supervision for Mathematical Reasoning in LLMs
Recent studies reveal that large language models (LLMs) exhibit limited logical reasoning abilities in mathematical problem-solving, instead often relying on pattern-matching and memorization. We systematically analyze t…
Mathematical ReasoningLogical ReasoningIntegrating BERT and Score-based Feature Gates for Chinese Grammatical Error Diagnosis
This paper describes our proposed model for the Chinese Grammatical Error Diagnosis (CGED) task in NLPTEA2020. The goal of CGED is to use natural language processing techniques to automatically diagnose Chinese grammatic…
Visual Error Patterns in Multi-Modal AI: A Statistical Approach
Multi-modal large language models (MLLMs), such as GPT-4o, excel at integrating text and visual data but face systematic challenges when interpreting ambiguous or incomplete visual stimuli. This study leverages statistic…
Feature Importance