Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study

2025-06-13 · Gábor Antal, Bence Bogenfürst, Rudolf Ferenc, Péter Hegedűs

Recent advancements in large language models (LLMs) have shown promise for automated vulnerability detection and repair in software systems. This paper investigates the performance of GPT-4o in repairing Java vulnerabilities from a widely used dataset (Vul4J), exploring how different contextual information affects automated vulnerability repair (AVR) capabilities. We compare the latest GPT-4o's performance against previous results with GPT-4 using identical prompts. We evaluated nine additional prompts crafted by us that contain various contextual information such as CWE or CVE information, and manually extracted code contexts. Each prompt was executed three times on 42 vulnerabilities, and the resulting fix candidates were validated using Vul4J's automated testing framework. Our results show that GPT-4o performed 11.9\% worse on average than GPT-4 with the same prompt, but was able to fix 10.5\% more distinct vulnerabilities in the three runs together. CVE information significantly improved repair rates, while the length of the task description had minimal impact. Combining CVE guidance with manually extracted code context resulted in the best performance. Using our \textsc{Top}-3 prompts together, GPT-4o repaired 26 (62\%) vulnerabilities at least once, outperforming both the original baseline (40\%) and its reproduction (45\%), suggesting that ensemble prompt strategies could improve vulnerability repair in zero-shot settings.

📄 PDF Abstract BibTeX arXiv:2506.11561

Code (0)

등록된 구현이 없습니다.

Tasks

Vulnerability Detection

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Transformer A Transformer is a model architecture that eschews recurrence and instead relies entirely on an [attention…

Using ML filters to help automated vulnerability repairs: when it helps and when it doesn't

2025-04-09 · Maria Camporese, Fabio Massacci

[Context:] The acceptance of candidate patches in automated program repair has been typically based on testing oracles. Testing requires typically a costly process of building the application while ML models can be used …

Program Repair Vulnerability Detection

Code Vulnerability Repair with Large Language Model using Context-Aware Prompt Tuning

2024-09-27 · Arshiya Khan, Guannan Liu, Xing Gao

Large Language Models (LLMs) have shown significant challenges in detecting and repairing vulnerable code, particularly when dealing with vulnerabilities involving multiple aspects, such as variables, code flows, and cod…

Language Modeling Language Modelling Large Language Model Vulnerability Detection

MemRepair: Hierarchical Memory for Agentic Repository-Level Vulnerability Repair

2026-05-17 · Simiao Liu, Li Zhang, Fang Liu, Xiaoli Lian 외 arxiv

Modern software ecosystems face a rapidly growing number of disclosed vulnerabilities, increasing the need for automated repair techniques that can operate reliably at repository scale. Although Large Language Model (LLM…

Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study

2026-06-24 · Giulian Biolo, Michael Tezza, Yuanjun Gong, Fabio Massacci arxiv

Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. In the meantime, Large Language Models (LLMs) assisted tools show poten…

Vulnerability Detection

CRepair: CVAE-based Automatic Vulnerability Repair Technology

2024-11-08 · Penghui Liu, Yingzhou Bi, Jiangtao Huang, Xinxin Jiang 외

Software vulnerabilities are flaws in computer software systems that pose significant threats to the integrity, security, and reliability of modern software and its application data. These vulnerabilities can lead to sub…

Causal Inference