FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (2) · 1087-1095
DOI: 10.1609/aaai.v40i2.37079 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Gwok-Waa Wan und 18 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref).
Abstract (Originalsprache)
We introduce FIXME, the first end-to-end and large-scale benchmark for evaluating Large Language Models (LLMs) in hardware design functional verification (FV). Comprising 747 tasks derived from real-world hardware designs, FIXME spans five core FV sub-sets: specification comprehension, reference model generation, testbench generation, assertion design, and RTL debugging. To ensure high data quality, we developed an AI-human collaborative framework for agile data curation and annotation. This process resulted in 25,000 lines of verified RTL, 35,000 lines of enhanced testbenches, and over 1,200 SystemVerilog Assertions. Furthermore, through expert-guided optimization within the multi-agent aided flow, we achieved a remarkable 45.57% improvement in average functional coverage, underscoring the benchmark's robustness. Through evaluation of state-of-the-art LLMs like GPT-4.1, FIXME identifies key limitations and provides actionable insights, advancing the potential of LLM-driven automation in hardware design functional verification.
Zitieren
APA
Wan, G., Wong, S., Su, S., Niu, C., Wang, N., Wan, X., Chen, Q., Xing, M., Zhang, J., Ye, J., Wang, Y., Song, R., Ni, T., Xu, Q., Guan, N., Jiang, Z., Wang, X., Chen, Y., & Yang, J. (2026). FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification. Proceedings of the AAAI Conference on Artificial Intelligence, 40(2), 1087-1095. https://doi.org/10.1609/aaai.v40i2.37079
Vancouver
Wan G, Wong S, Su S, Niu C, Wang N, Wan X, et al. FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(2):1087-1095. doi:10.1609/aaai.v40i2.37079
BibTeX
@article{Wan2026,
author = {Wan, Gwok-Waa and Wong, SamZaak and Su, Shengchu and Niu, Chenxu and Wang, Ning and Wan, Xinlai and Chen, Qixiang and Xing, Mengnv and Zhang, Jingyi and Ye, Jianmin and Wang, Yubo and Song, Rongchang and Ni, Tao and Xu, Qiang and Guan, Nan and Jiang, Zhe and Wang, Xi and Chen, Yong and Yang, Jun},
title = {FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {2},
pages = {1087--1095},
doi = {10.1609/aaai.v40i2.37079}
}
EUVIMED