EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

Model Checking Using Large Language Models—Evaluation and Future Directions

Sotiris Batsakis, Ilias Tachmazidis, Matthew Mantle, Nikolaos Papadakis, Grigoris Antoniou

Electronics · 2025-01-20 · 14 (2) · 401

DOI: 10.3390/electronics14020401 ↗

At a glance

Journal article in Electronics, published online on 2025-01-20. Written by Sotiris Batsakis and 4 co-authors. Cited 7 times so far (Crossref); 45 references. The full text is freely available (open access, CC BY 4.0).

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Large language models (LLMs) such as ChatGPT have risen in prominence recently, leading to the need to analyze their strengths and limitations for various tasks. The objective of this work was to evaluate the performance of large language models for model checking, which is used extensively in various critical tasks such as software and hardware verification. A set of problems were proposed as a benchmark in this work and three LLMs (GPT-4, Claude, and Gemini) were evaluated with respect to their ability to solve these problems. The evaluation was conducted by comparing the responses of the three LLMs with the gold standard provided by model checking tools. The results illustrate the limitations of LLMs in these tasks, identifying directions for future research. Specifically, the best overall performance (ratio of problems solved correctly) was 60%, indicating a high probability of reasoning errors by the LLMs, especially when dealing with more complex scenarios requiring many reasoning steps, and the LLMs typically performed better when generating scripts for solving the problems rather than solving them directly.

Cite

APA

Batsakis, S., Tachmazidis, I., Mantle, M., Papadakis, N., & Antoniou, G. (2025). Model Checking Using Large Language Models—Evaluation and Future Directions. Electronics, 14(2), 401. https://doi.org/10.3390/electronics14020401

Vancouver

Batsakis S, Tachmazidis I, Mantle M, Papadakis N, Antoniou G. Model Checking Using Large Language Models—Evaluation and Future Directions. Electronics. 2025;14(2):401. doi:10.3390/electronics14020401

BibTeX
@article{Batsakis2025,
  author = {Batsakis, Sotiris and Tachmazidis, Ilias and Mantle, Matthew and Papadakis, Nikolaos and Antoniou, Grigoris},
  title = {Model Checking Using Large Language Models—Evaluation and Future Directions},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {2},
  pages = {401},
  doi = {10.3390/electronics14020401}
}
Start for free →