Leveraging Large Language Models in Code Question Answering: Baselines and Issues
Question answering over source code provides software engineers and project managers with helpful information about the implemented features of a software product. This paper presents a work devoted to using large language models for question answering over source code in Python. The proposed method for implementing a source code question answering system involves fine-tuning a large language model on a unified dataset of questions and answers for Python code. To achieve the highest quality answers, we tested various models trained on datasets preprocessed in different ways: a dataset without grammar correction, a dataset with grammar correction, and a dataset augmented with the generated summaries. The model answers were also analyzed for errors manually. We report BLEU-4, BERTScore F1, BLEURT, and Exact Match metric values, along with the conclusions from the manual error analysis. The obtained experimental results highlight the current problems of the research area, such as poor quality of the public genuine question-answering datasets. In addition, the findings include the positive effect of the grammar correction of the training data on the testing metric values. The addressed findings and issues could be important for other researchers who attempt to improve the quality of source code question answering solutions. The training and evaluation code is publicly available at https://github.com/IU-AES-AI4Code/CodeQuestionAnswering.
Code (1)
Tasks
Large Language ModelQuestion AnsweringSimilar Papers 제목 키워드 기반
Leveraging LLMs in Scholarly Knowledge Graph Question Answering
This paper presents a scholarly Knowledge Graph Question Answering (KGQA) that answers bibliographic natural language questions by leveraging a large language model (LLM) in a few-shot manner. The model initially identif…
Graph Question AnsweringLanguage ModelingLanguage ModellingLarge Language Model+2UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models
Generative information retrieval, encompassing two major tasks of Generative Document Retrieval (GDR) and Grounded Answer Generation (GAR), has gained significant attention in the area of information retrieval and natura…
Answer GenerationInformation RetrievalQuestion AnsweringRetrievalThe Role of Exploration Modules in Small Language Models for Knowledge Graph Question Answering
Integrating knowledge graphs (KGs) into the reasoning processes of large language models (LLMs) has emerged as a promising approach to mitigate hallucination. However, existing work in this area often relies on proprieta…
Graph Question AnsweringKnowledge GraphsQuestion Answering as Programming for Solving Time-Sensitive Questions
Question answering plays a pivotal role in human daily life because it involves our acquisition of knowledge about the world. However, due to the dynamic and ever-changing nature of real-world facts, the answer can be co…
Natural Language UnderstandingQuestion AnsweringLifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos
In this paper we introduce LifelongMemory, a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. LifelongMemory generates concise video activity…
EgoSchemaFormQuestion AnsweringRetrieval