SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
The code generation capabilities of large language models(LLMs) have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark specifically designed to assess the security of LLM-generated code. The dataset encompasses a wide range of common software development scenarios and vulnerability types. Building upon this benchmark, we develop an automatic evaluation framework that leverages both static application security testing(SAST) and LLM-based judging to assess the presence of security vulnerabilities in model-generated code. Through the empirical evaluation of state-of-the-art LLMs on SafeGenBench, we reveal notable deficiencies in their ability to produce vulnerability-free code. Our findings highlight pressing challenges and offer actionable insights for future advancements in the secure code generation performance of LLMs. The data and code will be released soon.
Code (0)
등록된 구현이 없습니다.
Tasks
Code GenerationVulnerability DetectionSimilar Papers 제목 키워드 기반
SoK: Understanding (New) Security Issues Across AI4Code Use Cases
AI-for-Code (AI4Code) systems are reshaping software engineering, with tools like GitHub Copilot accelerating code generation, translation, and vulnerability detection. Alongside these advances, however, security risks r…
Vulnerability DetectionAdversarial RobustnessCode TranslationCode GenerationCo-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents
Large language models (LLMs) have shown promise in assisting cybersecurity tasks, yet existing approaches struggle with automatic vulnerability discovery and exploitation due to limited interaction, weak execution ground…
Vulnerability DetectionToward Scalable Automated Repository-Level Datasets for Software Vulnerability Detection
Software vulnerabilities continue to grow in volume and remain difficult to detect in practice. Although learning-based vulnerability detection has progressed, existing benchmarks are largely function-centric and fail to…
Vulnerability DetectionHow Far Have We Gone in Vulnerability Detection Using Large Language Models
As software becomes increasingly complex and prone to vulnerabilities, automated vulnerability detection is critically important, yet challenging. Given the significant successes of large language models (LLMs) in variou…
Vulnerability DetectionBeyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection
Large Language Models (LLMs) have demonstrated significant potential in automated software security, particularly in vulnerability detection. However, existing benchmarks primarily focus on isolated, single-vulnerability…
Vulnerability Detection