Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation
ACM Transactions on Design Automation of Electronic Systems · 30.11.2025 · 30 (6) · 1-20
DOI: 10.1145/3718088 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Design Automation of Electronic Systems, online erschienen am 21.10.2025. Verfasst von Nathaniel Pinckney und 4 weiteren Autor:innen. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 37 Literaturangaben.
Abstract (Originalsprache)
The application of large language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval’s original release—including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder—against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval’s infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.
Zitieren
APA
Pinckney, N., Batten, C., Liu, M., Ren, H., & Khailany, B. (2025). Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation. ACM Transactions on Design Automation of Electronic Systems, 30(6), 1-20. https://doi.org/10.1145/3718088
Vancouver
Pinckney N, Batten C, Liu M, Ren H, Khailany B. Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation. ACM Transactions on Design Automation of Electronic Systems. 2025;30(6):1-20. doi:10.1145/3718088
BibTeX
@article{Pinckney2025,
author = {Pinckney, Nathaniel and Batten, Christopher and Liu, Mingjie and Ren, Haoxing and Khailany, Brucek},
title = {Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation},
journal = {ACM Transactions on Design Automation of Electronic Systems},
year = {2025},
volume = {30},
number = {6},
pages = {1--20},
doi = {10.1145/3718088}
}
EUVIMED