EUVIMEDEUVIMED entdecken
Crossref

FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification

Gwok-Waa Wan, SamZaak Wong, Shengchu Su, Chenxu Niu, Ning Wang, Xinlai Wan, Qixiang Chen, Mengnv Xing, Jingyi Zhang, Jianmin Ye, Yubo Wang, Rongchang Song und 7 weitere

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (2) · 1087-1095

DOI: 10.1609/aaai.v40i2.37079 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Gwok-Waa Wan und 18 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce FIXME, the first end-to-end and large-scale benchmark for evaluating Large Language Models (LLMs) in hardware design functional verification (FV). Comprising 747 tasks derived from real-world hardware designs, FIXME spans five core FV sub-sets: specification comprehension, reference model generation, testbench generation, assertion design, and RTL debugging. To ensure high data quality, we developed an AI-human collaborative framework for agile data curation and annotation. This process resulted in 25,000 lines of verified RTL, 35,000 lines of enhanced testbenches, and over 1,200 SystemVerilog Assertions. Furthermore, through expert-guided optimization within the multi-agent aided flow, we achieved a remarkable 45.57% improvement in average functional coverage, underscoring the benchmark's robustness. Through evaluation of state-of-the-art LLMs like GPT-4.1, FIXME identifies key limitations and provides actionable insights, advancing the potential of LLM-driven automation in hardware design functional verification.

Zitieren

APA

Wan, G., Wong, S., Su, S., Niu, C., Wang, N., Wan, X., Chen, Q., Xing, M., Zhang, J., Ye, J., Wang, Y., Song, R., Ni, T., Xu, Q., Guan, N., Jiang, Z., Wang, X., Chen, Y., & Yang, J. (2026). FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification. Proceedings of the AAAI Conference on Artificial Intelligence, 40(2), 1087-1095. https://doi.org/10.1609/aaai.v40i2.37079

Vancouver

Wan G, Wong S, Su S, Niu C, Wang N, Wan X, et al. FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(2):1087-1095. doi:10.1609/aaai.v40i2.37079

BibTeX
@article{Wan2026,
  author = {Wan, Gwok-Waa and Wong, SamZaak and Su, Shengchu and Niu, Chenxu and Wang, Ning and Wan, Xinlai and Chen, Qixiang and Xing, Mengnv and Zhang, Jingyi and Ye, Jianmin and Wang, Yubo and Song, Rongchang and Ni, Tao and Xu, Qiang and Guan, Nan and Jiang, Zhe and Wang, Xi and Chen, Yong and Yang, Jun},
  title = {FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {2},
  pages = {1087--1095},
  doi = {10.1609/aaai.v40i2.37079}
}
Kostenlos starten →