SyllabusQA: A Course Logistics Question Answering Dataset
Automated teaching assistants and chatbots have significant potential to reduce the workload of human instructors, especially for logistics-related question answering, which is important to students yet repetitive for instructors. However, due to privacy concerns, there is a lack of publicly available datasets. We introduce SyllabusQA, an open-source dataset with 63 real course syllabi covering 36 majors, containing 5,078 open-ended course logistics-related question-answer pairs that are diverse in both question types and answer formats. Since many logistics-related questions contain critical information like the date of an exam, it is important to evaluate the factuality of answers. We benchmark several strong baselines on this task, from large language model prompting to retrieval-augmented generation. We introduce Fact-QA, an LLM-based (GPT-4) evaluation metric to evaluate the factuality of predicted answers. We find that despite performing close to humans on traditional metrics of textual similarity, there remains a significant gap between automated approaches and humans in terms of fact precision.
Code (1)
Tasks
Language ModelingLanguage ModellingLarge Language ModelQuestion AnsweringRetrievalRetrieval-augmented GenerationSimilar Papers 제목 키워드 기반
Decomposed Prompting to Answer Questions on a Course Discussion Board
We propose and evaluate a question-answering system that uses decomposed prompting to classify and answer student questions on a course discussion board. Our system uses a large language model (LLM) to classify questions…
Language ModelingLanguage ModellingLarge Language ModelQuestion AnsweringAnswering Students' Questions on Course Forums Using Multiple Chain-of-Thought Reasoning and Finetuning RAG-Enabled LLM
The course forums are increasingly significant and play vital role in facilitating student discussions and answering their questions related to the course. It provides a platform for students to post their questions rela…
Question AnsweringPlatform-Independent and Curriculum-Oriented Intelligent Assistant for Higher Education
Miscommunication and communication challenges between instructors and students represents one of the primary barriers to post-secondary learning. Students often avoid or miss opportunities to ask questions during office …
Language ModellingSchedulingDiscourse Comprehension: A Question Answering Framework to Represent Sentence Connections
While there has been substantial progress in text comprehension through simple factoid question answering, more holistic comprehension of a discourse still presents a major challenge (Dunietz et al., 2020). Someone criti…
Question AnsweringReading ComprehensionSentenceDiscourse-Based Approach to Involvement of Background Knowledge for Question Answering
We introduce a concept of a virtual discourse tree to improve question answering (Q/A) recall for complex, multi-sentence questions. Augmenting the discourse tree of an answer with tree fragments obtained from text corpo…
Question AnsweringSentence