Generative AI for Software Metadata: Overview of the Information Retrieval in Software Engineering Track at FIRE 2023
The Information Retrieval in Software Engineering (IRSE) track aims to develop solutions for automated evaluation of code comments in a machine learning framework based on human and large language model generated labels. In this track, there is a binary classification task to classify comments as useful and not useful. The dataset consists of 9048 code comments and surrounding code snippet pairs extracted from open source github C based projects and an additional dataset generated individually by teams using large language models. Overall 56 experiments have been submitted by 17 teams from various universities and software companies. The submissions have been evaluated quantitatively using the F1-Score and qualitatively based on the type of features developed, the supervised learning model used and their corresponding hyper-parameters. The labels generated from large language models increase the bias in the prediction model but lead to less over-fitted results.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationInformation RetrievalLanguage ModelingLanguage ModellingLarge Language ModelRetrievalSimilar Papers 제목 키워드 기반
Building a research-software catalog with a coding agent: from hackathon prototype to public deployment
Generative AI and coding agents can accelerate research software development, but they also increase the need for efficient software discovery and maintenance. We developed a repository catalog during a three-day hackath…
S3LLM: Large-Scale Scientific Software Understanding with LLMs using Source, Metadata, and Document
The understanding of large-scale scientific software poses significant challenges due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large…
Natural Language QueriesRAGRetrieval-augmented GenerationUser Manual of Automatic Data Curation Tool(ADCT): A bulk data curator software in Library and Information Science
In library and information science, document storage and user-specific document retrieval are the main aspects of digital library services. To preserve the cultural heritage, documents, and literature, we need a common p…
DescriptiveRetrievalTREC 2020 Podcasts Track Overview
The Podcast Track is new at the Text Retrieval Conference (TREC) in 2020. The podcast track was designed to encourage research into podcasts in the information retrieval and NLP research communities. The track consisted …
Information RetrievalRetrievalText RetrievalOverview of the TREC 2023 Product Product Search Track
This is the first year of the TREC Product search track. The focus this year was the creation of a reusable collection and evaluation of the impact of the use of metadata and multi-modal data on retrieval accuracy. This …
Retrieval