VerilogEval: Evaluating Large Language Models for Verilog Code Generation

2023-09-14 · Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing Ren

The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.

📄 PDF Abstract BibTeX arXiv:2309.07544

Code (1)

nvlabs/verilog-eval 공식 구현

Tasks

Benchmarking Code Generation

Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation

2024-08-20 · Nathaniel Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren 외

The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a s…

Code Completion Code Generation In-Context Learning Prompt Engineering

Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback

2025-04-22 · Ning Wang, Bingkun Yao, Jie zhou, Yuchen Hu 외

Large language models (LLMs) have shown strong performance in Verilog generation from natural language description. However, ensuring the functional correctness of the generated code remains a significant challenge. This…

Code Generation Hallucination Reinforcement Learning (RL)

CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

2024-09-19 · Mingjie Liu, Yun-Da Tsai, Wenfei Zhou, Haoxing Ren

Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned…

Code Generation Code Repair Synthetic Data Generation

hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation

2025-06-05 · Charles Hong, Brendan Roberts, Huijae An, Alex Um 외

Large language models (LLMs) are playing an increasingly large role in domains such as code generation, including hardware code generation, where Verilog is the key language. However, the amount of publicly available Ver…

Code Generation Code Translation Data Augmentation Knowledge Distillation

VeriGRAG: Enhancing LLM-Based Verilog Code Generation with Structure-Aware Soft Prompts

2025-09-27 · Jiayu Zhao, Song Chen arxiv

Large language models (LLMs) have demonstrated strong capabilities in generating Verilog code from natural language descriptions. However, Verilog code inherently encodes structural information of hardware circuits. Effe…

Code Generation