홈 › Papers

Enhancing Programming Error Messages in Real Time with Generative AI

2024-02-12 · Bailey Kimmel, Austin Geisert, Lily Yaro, Brendan Gipson, Taylor Hotchkiss, Sidney Osae-Asante, Hunter Vaught, Grant Wininger, Chase Yamaguchi

Generative AI is changing the way that many disciplines are taught, including computer science. Researchers have shown that generative AI tools are capable of solving programming problems, writing extensive blocks of code, and explaining complex code in simple terms. Particular promise has been shown in using generative AI to enhance programming error messages. Both students and instructors have complained for decades that these messages are often cryptic and difficult to understand. Yet recent work has shown that students make fewer repeated errors when enhanced via GPT-4. We extend this work by implementing feedback from ChatGPT for all programs submitted to our automated assessment tool, Athene, providing help for compiler, run-time, and logic errors. Our results indicate that adding generative AI to an automated assessment tool does not necessarily make it better and that design of the interface matters greatly to the usability of the feedback that GPT-4 provided.

📄 PDF Abstract BibTeX arXiv:2402.08072

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Using Large Language Models to Enhance Programming Error Messages

2022-10-20 · Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves 외

A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the me…

Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice

2024-09-27 · Eddie Antonio Santos, Brett A. Becker

The sudden emergence of large language models (LLMs) such as ChatGPT has had a disruptive impact throughout the computing education community. LLMs have been shown to excel at producing correct code to CS1 and CS2 proble…

Unlocking LLM Code Correction with Iterative Feedback Loops

2026-06-16 · Le Zhang, Suresh Kothari arxiv

Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-…

Code Generation

DeepFix: Fixing Common C Language Errors by Deep Learning

2017-02-04 · Rahul Gupta, Soham Pal, Aditya Kanade, Shirish Shevade

The problem of automatically fixing programming errors is a very active research topic in software engineering. This is a challenging problem as fixing even a single error may require analysis of the entire program. In p…

Deep Learning Program Repair

SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics

2021-04-29 · Toufique Ahmed, Noah Rose Ledesma, Premkumar Devanbu

Beginning programmers struggle with the complex grammar of modern programming languages like Java, and make lot of syntax errors. The diagnostic syntax error messages from compilers and IDEs are sometimes useful, but oft…

BIG-bench Machine Learning Diagnostic Multi-Label Classification MUlTI-LABEL-ClASSIFICATION+1