EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Model Checking Using Large Language Models—Evaluation and Future Directions

Sotiris Batsakis, Ilias Tachmazidis, Matthew Mantle, Nikolaos Papadakis, Grigoris Antoniou

Electronics · 20.01.2025 · 14 (2) · 401

DOI: 10.3390/electronics14020401 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 20.01.2025. Verfasst von Sotiris Batsakis und 4 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) such as ChatGPT have risen in prominence recently, leading to the need to analyze their strengths and limitations for various tasks. The objective of this work was to evaluate the performance of large language models for model checking, which is used extensively in various critical tasks such as software and hardware verification. A set of problems were proposed as a benchmark in this work and three LLMs (GPT-4, Claude, and Gemini) were evaluated with respect to their ability to solve these problems. The evaluation was conducted by comparing the responses of the three LLMs with the gold standard provided by model checking tools. The results illustrate the limitations of LLMs in these tasks, identifying directions for future research. Specifically, the best overall performance (ratio of problems solved correctly) was 60%, indicating a high probability of reasoning errors by the LLMs, especially when dealing with more complex scenarios requiring many reasoning steps, and the LLMs typically performed better when generating scripts for solving the problems rather than solving them directly.

Zitieren

APA

Batsakis, S., Tachmazidis, I., Mantle, M., Papadakis, N., & Antoniou, G. (2025). Model Checking Using Large Language Models—Evaluation and Future Directions. Electronics, 14(2), 401. https://doi.org/10.3390/electronics14020401

Vancouver

Batsakis S, Tachmazidis I, Mantle M, Papadakis N, Antoniou G. Model Checking Using Large Language Models—Evaluation and Future Directions. Electronics. 2025;14(2):401. doi:10.3390/electronics14020401

BibTeX
@article{Batsakis2025,
  author = {Batsakis, Sotiris and Tachmazidis, Ilias and Mantle, Matthew and Papadakis, Nikolaos and Antoniou, Grigoris},
  title = {Model Checking Using Large Language Models—Evaluation and Future Directions},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {2},
  pages = {401},
  doi = {10.3390/electronics14020401}
}
Kostenlos starten →