EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Do LLMs Speak BPMN? An Evaluation of Their Process Modeling Capabilities Based on Quality Measures

Panagiotis Drakopoulos, Panagiotis Malousoudis, Nikolaos Nousias, George Tsakalidis, Kostas Vergidis

Computation · 04.01.2026 · 14 (1) · 10

DOI: 10.3390/computation14010010 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computation, online erschienen am 04.01.2026. Verfasst von Panagiotis Drakopoulos und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) are emerging as powerful tools for automating business process modeling, promising to streamline the translation of textual process descriptions into Business Process Model and Notation (BPMN) diagrams. However, the extent to which these Al systems can produce high-quality BPMN models has not yet been rigorously evaluated. This paper presents an early evaluation of five LLM-powered BPMN generation tools that automatically convert textual process descriptions into BPMN models. To assess the external quality of these Al-generated models, we introduce a novel structured evaluation framework that scores each BPMN diagram across three key process model quality dimensions: clarity, correctness, and completeness, covering both accuracy and diagram understandability. Using this framework, we conducted experiments where each tool was tasked with modeling the same set of textual process scenarios, and the resulting diagrams were systematically scored based on the criteria. This approach provides a consistent and repeatable evaluation procedure and offers a new lens for comparing LLM-based modeling capabilities. Given the focused scope of the study, the results should be interpreted as an exploratory benchmark that surfaces initial observations about tool performance rather than definitive conclusions. Our findings reveal that while current LLM-based tools can produce BPMN diagrams that capture the main elements of a process description, they often exhibit errors such as missing steps, inconsistent logic, or modeling rule violations, highlighting limitations in achieving fully correct and complete models. The clarity and readability of the generated diagrams also vary, indicating that these Al models are still maturing in generating easily interpretable process flows. We conclude that although LLMs show promise in automating BPMN modeling, significant improvements are needed for them to consistently generate both syntactically and semantically valid process models.

Zitieren

APA

Drakopoulos, P., Malousoudis, P., Nousias, N., Tsakalidis, G., & Vergidis, K. (2026). Do LLMs Speak BPMN? An Evaluation of Their Process Modeling Capabilities Based on Quality Measures. Computation, 14(1), 10. https://doi.org/10.3390/computation14010010

Vancouver

Drakopoulos P, Malousoudis P, Nousias N, Tsakalidis G, Vergidis K. Do LLMs Speak BPMN? An Evaluation of Their Process Modeling Capabilities Based on Quality Measures. Computation. 2026;14(1):10. doi:10.3390/computation14010010

BibTeX
@article{Drakopoulos2026,
  author = {Drakopoulos, Panagiotis and Malousoudis, Panagiotis and Nousias, Nikolaos and Tsakalidis, George and Vergidis, Kostas},
  title = {Do LLMs Speak BPMN? An Evaluation of Their Process Modeling Capabilities Based on Quality Measures},
  journal = {Computation},
  year = {2026},
  volume = {14},
  number = {1},
  pages = {10},
  doi = {10.3390/computation14010010}
}
Kostenlos starten →