CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course
We introduce CS1QA, a dataset for code-based question answering in the programming education domain. CS1QA consists of 9,237 question-answer pairs gathered from chat logs in an introductory programming class using Python, and 17,698 unannotated chat data with code. Each question is accompanied with the student's code, and the portion of the code relevant to answering the question. We carefully design the annotation process to construct CS1QA, and analyze the collected dataset in detail. The tasks for CS1QA are to predict the question type, the relevant code snippet given the question and the code and retrieving an answer from the annotated corpus. Results for the experiments on several baseline models are reported and thoroughly analyzed. The tasks for CS1QA challenge models to understand both the code and natural language. This unique dataset can be used as a benchmark for source code comprehension and question answering in the educational setting.
Code (1)
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
Analyzing ChatGPT's Aptitude in an Introductory Computer Engineering Course
ChatGPT has recently gathered attention from the general public and academia as a tool that is able to generate plausible and human-sounding text answers to various questions. One potential use, or abuse, of ChatGPT is i…
Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses
In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-trac…
Perhaps PTLMs Should Go to School – A Task to Assess Open Book and Closed Book QA
Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college text…
Question AnsweringPerhaps PTLMs Should Go to School -- A Task to Assess Open Book and Closed Book QA
Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college text…
Question AnsweringCSEPrompts: A Benchmark of Introductory Computer Science Prompts
Recent advances in AI, machine learning, and NLP have led to the development of a new generation of Large Language Models (LLMs) that are trained on massive amounts of data and often have trillions of parameters. Commerc…
Multiple-choice