홈 › Papers

HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

2026-01-20 · Qirui Chen, Jingxian Shuai, Shuangwu Chen, Shenghao Ye, Zijian Wen, Xufei Su, Jie Jin, Jiangming Li, Jun Chen, Xiaobin Tan, Jian Yang arxiv

Large language models (LLMs) are increasingly used for hardware and firmware code generation, but existing studies primarily evaluate functional correctness while largely overlooking security. However, LLM-generated code that appears functionally sound may embed security flaws which could induce catastrophic damages after deployment. This critical research gap motivates us to design a benchmark for assessing security awareness under realistic specifications. In this work, we introduce HardSecBench, a benchmark with 924 tasks spanning Verilog Register Transfer Level (RTL) and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation, and executable tests. To automate artifact synthesis, we propose a multi-agent pipeline that decouples synthesis from verification and grounds evaluation in execution evidence, enabling reliable evaluation. We evaluate diverse LLMs and find that they often satisfy functional requirements while leaving security risks. We also find that security results vary with prompting. These findings highlight pressing challenges and offer actionable insights for future advancements in LLM-assisted hardware design. Our data and code are available at https://github.com/chenqirui2002/HardSecBench.

📄 PDF Abstract BibTeX arXiv:2601.13864

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation

LLMs for Secure Hardware Design and Related Problems: Opportunities and Challenges

2026-05-11 · Johann Knechtel, Ozgur Sinanoglu, Ramesh Karri arxiv

The integration of Large Language Models (LLMs) into Electronic Design Automation (EDA) and hardware security is rapidly reshaping the semiconductor industry. While LLMs offer unprecedented capabilities in generating Reg…

ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness

2025-03-11 · Ce Guo, Tong Zhao

Field-Programmable Gate Arrays (FPGAs) are widely used in modern hardware design, yet writing Hardware Description Language (HDL) code for FPGA implementation remains a complex and time-consuming task. Large Language Mod…

Benchmarking Code Generation Diversity

KernelFoundry: Hardware-aware evolutionary GPU kernel optimization

2026-03-12 · Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk 외 arxiv

Optimizing GPU kernels presents a significantly greater challenge for large language models (LLMs) than standard code generation tasks, as it requires understanding hardware architecture, parallel optimization strategies…

Code Generation

I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench

2024-01-31 · Yuan Li, Yue Huang, Yuli Lin, Siyuan Wu 외

Do large language models (LLMs) exhibit any forms of awareness similar to humans? In this paper, we introduce AwareBench, a benchmark designed to evaluate awareness in LLMs. Drawing from theories in psychology and philos…

Benchmarking Multiple-choice Philosophy

LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators

2024-10-31 · Krishna Teja Chitty-Venkata, Siddhisanket Raskar, Bharat Kale, Farah Ferdaus 외

Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. However, the computational demands of these complex models pose signi…

Benchmarking Text Generation