VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection

2022-05-25 · Hazim Hanif, Sergio Maffeis

This paper presents VulBERTa, a deep learning approach to detect security vulnerabilities in source code. Our approach pre-trains a RoBERTa model with a custom tokenisation pipeline on real-world code from open-source C/C++ projects. The model learns a deep knowledge representation of the code syntax and semantics, which we leverage to train vulnerability detection classifiers. We evaluate our approach on binary and multi-class vulnerability detection tasks across several datasets (Vuldeepecker, Draper, REVEAL and muVuldeepecker) and benchmarks (CodeXGLUE and D2A). The evaluation results show that VulBERTa achieves state-of-the-art performance and outperforms existing approaches across different datasets, despite its conceptual simplicity, and limited cost in terms of size of training data and number of model parameters.

📄 PDF Abstract BibTeX arXiv:2205.12424

Code (1)

icl-ml4csec/vulberta 공식 구현 pytorch

Tasks

Vulnerability Detection

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Weight Decay 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection

2025-12-11 · Chaomeng Lu, Bert Lagaisse arxiv

Vulnerability detection methods based on deep learning (DL) have shown strong performance on benchmark datasets, yet their real-world effectiveness remains underexplored. Recent work suggests that graph neural network-ba…

Vulnerability Detection Graph Neural Network

Finetuning Large Language Models for Vulnerability Detection

2024-01-30 · Alexey Shestov, Rodion Levichev, Ravil Mussabayev, Evgeny Maslov 외

This paper presents the results of finetuning large language models (LLMs) for the task of detecting vulnerabilities in source code. We leverage WizardCoder, a recent improvement of the state-of-the-art LLM StarCoder, an…

Transfer Learning Vulnerability Detection

The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs

2025-04-16 · Haonan Li, Hang Zhang, Kexin Pei, Zhiyun Qian

Static analysis plays a crucial role in software vulnerability detection, yet faces a persistent precision-scalability tradeoff. In large codebases like the Linux kernel, traditional static analysis tools often generate …

Vulnerability Detection

To Err is Machine: Vulnerability Detection Challenges LLM Reasoning

2024-03-25 · Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam 외

In this paper, we present a challenging code reasoning task: vulnerability detection. Large Language Models (LLMs) have shown promising results in natural-language and math reasoning, but state-of-the-art (SOTA) models r…

Code Generation In-Context Learning Math Program Repair+1

DCDetector: An IoT terminal vulnerability mining system based on distributed deep ensemble learning under source code representation

2022-11-29 · Wen Zhou

Context: The IoT system infrastructure platform facility vulnerability attack has become the main battlefield of network security attacks. Most of the traditional vulnerability mining methods rely on vulnerability detect…

Ensemble Learning Sentence Vulnerability Detection