A comparison of Human, GPT-3.5, and GPT-4 Performance in a University-Level Coding Course
This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed $n = 300$ data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8) - a statistically significant difference (p = $2.482 \times 10^{-10}$). Prompt engineering significantly improved scores for both GPT-4 (p = $1.661 \times 10^{-4}$) and GPT-3.5 (p = $4.967 \times 10^{-9}$). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from Definitely AI' to Definitely Human'. They accurately identified the authorship, with 92.1% of the work categorized as 'Definitely Human' being human-authored. Simplifying this to a binary AI' or Human' categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students' work, it often remains detectable by human evaluators.
Code (1)
Tasks
Prompt EngineeringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
JU-Saarland Submission to the WMT2019 English--Gujarati Translation Shared Task
In this paper we describe our joint submission (JU-Saarland) from Jadavpur University and Saarland University in the WMT 2019 news translation shared task for English{--}Gujarati language pair within the translation task…
Machine TranslationNMTTranslationThe University of Helsinki Submissions to the WMT19 Similar Language Translation Task
This paper describes the University of Helsinki Language Technology group{'}s participation in the WMT 2019 similar language translation task. We trained neural machine translation models for the language pairs Czech {\t…
Machine TranslationSegmentationTranslationSciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading
With the rapid development of Large Language Models (LLMs), it is crucial to have benchmarks which can evaluate the ability of LLMs on different domains. One common use of LLMs is performing tasks on scientific topics, s…
BenchmarkingMathematical ProofsKnockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or …
Machine TranslationA Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
We demonstrate that a neural network pre-trained on text and fine-tuned on code solves mathematics course problems, explains solutions, and generates new questions at a human level. We automatically synthesize programs u…
Few-Shot LearningLanguage ModellingMathMathematical Reasoning+2