An Evalutation of Programming Language Models' performance on Software Defect Detection
This dissertation presents an evaluation of several language models on software defect datasets. A language Model (LM) "can provide word representation and probability indication of word sequences as the core component of an NLP system." Language models for source code are specified for tasks in the software engineering field. While some models are directly the NLP ones, others contain structural information that is uniquely owned by source code. Software defects are defects in the source code that lead to unexpected behaviours and malfunctions at all levels. This study provides an original attempt to detect these defects at three different levels (syntactical, algorithmic and general) We also provide a tool chain that researchers can use to reproduce the experiments. We have tested the different models against different datasets, and performed an analysis over the results. Our original attempt to deploy bert, the state-of-the-art model for multitasks, leveled or outscored all other models compared.
Code (1)
Tasks
Defect DetectionLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
v-SVR Polynomial Kernel for Predicting the Defect Density in New Software Projects
An important product measure to determine the effectiveness of software processes is the defect density (DD). In this study, we propose the application of support vector regression (SVR) to predict the DD of new software…
BenchmarkingregressionNatural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
This paper provides a comprehensive review of the literature concerning the utilization of Natural Language Processing (NLP) techniques, with a particular focus on transformer-based large language models (LLMs) trained u…
Clone DetectionCode CompletionCode GenerationCode Summarization+3HEDP: A Method for Early Forecasting Software Defects based on Human Error Mechanisms
As the primary cause of software defects, human error is the key to understanding, and perhaps to predicting and avoiding them. Little research has been done to predict defects on the basis of the cognitive errors that c…
A Taxonomy of Prompt Defects in LLM Systems
Large Language Models (LLMs) have become key components of modern software, with prompts acting as their de-facto programming interface. However, prompt design remains largely empirical and small mistakes can cascade int…
Prompt EngineeringFrom Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work e…