Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback
ACM Transactions on Design Automation of Electronic Systems · 30.11.2025 · 30 (6) · 1-26
DOI: 10.1145/3723876 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Design Automation of Electronic Systems, online erschienen am 22.10.2025. Verfasst von Jason Blocklove und 5 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: Graduate Assistance in Areas of National Need.
Abstract (Originalsprache)
Traditionally, digital hardware designs are written in the Verilog hardware description language (HDL) and debugged manually by engineers. This can be time-consuming and error-prone for complex designs. Large Language Models (LLMs) are emerging as a potential tool to help generate fully functioning HDL code, but most works have focused on generation in the single-shot capacity: i.e., run and evaluate, a process that does not leverage debugging and, as such, does not adequately reflect a realistic development process. In this work, we evaluate the ability of LLMs to leverage feedback from electronic design automation (EDA) tools to fix mistakes in their own generated Verilog. To accomplish this, we present an open-source, highly customizable framework, AutoChip, which combines conversational LLMs with the output from Verilog compilers and simulations to iteratively generate and repair Verilog. To determine the success of these LLMs we leverage the VerilogEval benchmark set. We evaluate four state-of-the-art conversational LLMs, focusing on readily accessible commercial models. EDA tool feedback proved to be consistently more effective than zero-shot prompting only with GPT-4o, the most computationally complex model we evaluated. In the best case, we observed a 5.8% increase in the number of successful designs with a 34.2% decrease in cost over the best zero-shot results. Mixing smaller models with this larger model at the end of the feedback iterations resulted in equally as much success as with GPT-4o using feedback, but incurred 41.9% lower cost (corresponding to an overall decrease in cost over zero-shot by 89.6%).
Förderung
- Graduate Assistance in Areas of National Need
Zitieren
APA
Blocklove, J., Thakur, S., Tan, B., Pearce, H., Garg, S., & Karri, R. (2025). Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback. ACM Transactions on Design Automation of Electronic Systems, 30(6), 1-26. https://doi.org/10.1145/3723876
Vancouver
Blocklove J, Thakur S, Tan B, Pearce H, Garg S, Karri R. Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback. ACM Transactions on Design Automation of Electronic Systems. 2025;30(6):1-26. doi:10.1145/3723876
BibTeX
@article{Blocklove2025,
author = {Blocklove, Jason and Thakur, Shailja and Tan, Benjamin and Pearce, Hammond and Garg, Siddharth and Karri, Ramesh},
title = {Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback},
journal = {ACM Transactions on Design Automation of Electronic Systems},
year = {2025},
volume = {30},
number = {6},
pages = {1--26},
doi = {10.1145/3723876}
}
EUVIMED