EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Quality Assessment of Generative AI in Cybersecurity Certification

Vanessa G. Félix, Rodolfo Ostos, Luis J. Mena, Homero Toral-Cruz, Alberto Ochoa-Brust, Pablo Velarde-Alvarado, Apolinar González-Potes, Ramón A. Félix-Cuadras, José A. León-Borges, Rafael Martínez-Peláez

Informatics · 30.03.2026 · 13 (4) · 53

DOI: 10.3390/informatics13040053 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Informatics, online erschienen am 30.03.2026. Verfasst von Vanessa G. Félix und 9 weiteren Autor:innen. Die Arbeit nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: UNIVERSIDAD CATOLICA DEL NORTE https://ror.org/02akpm128.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Generative Artificial Intelligence (GenAI), particularly Large Language Models (LLMs), is rapidly changing how higher education approaches teaching, learning, and assessment. In cybersecurity education, professional certification exams are key for measuring competence and helping professionals find better job offers, but there is little research on how GenAI systems perform in these exam settings. This study looks at how three popular LLMs, ChatGPT-5, Gemini-2.5 Pro, and Copilot-2.5 Pro, handle 183 practice questions from the CompTIA Security+ certification. The study used a two-phase evaluation: a domain-based assessment and a full-length practice exam that mirrors real certification tests. The researchers measured model performance with accuracy scores, chi-square tests for statistical differences, and an error taxonomy to spot patterns of mistakes important for education. All three GenAI systems scored above the passing mark, and there were no significant differences between them. Still, the error analysis showed ongoing conceptual and classification mistakes that did not show up in the overall accuracy scores. Our results show that GenAI systems can pass structured certification tests, but accuracy by itself does not fully measure professional skills. The study points out important issues for the reliability and validity of AI-based assessments in higher education and stresses the need for more realistic, concept-focused ways to evaluate GenAI in cybersecurity education.

Förderung

  • UNIVERSIDAD CATOLICA DEL NORTE https://ror.org/02akpm128

Zitieren

APA

Félix, V. G., Ostos, R., Mena, L. J., Toral-Cruz, H., Ochoa-Brust, A., Velarde-Alvarado, P., González-Potes, A., Félix-Cuadras, R. A., León-Borges, J. A., & Martínez-Peláez, R. (2026). Quality Assessment of Generative AI in Cybersecurity Certification. Informatics, 13(4), 53. https://doi.org/10.3390/informatics13040053

Vancouver

Félix VG, Ostos R, Mena LJ, Toral-Cruz H, Ochoa-Brust A, Velarde-Alvarado P, et al. Quality Assessment of Generative AI in Cybersecurity Certification. Informatics. 2026;13(4):53. doi:10.3390/informatics13040053

BibTeX
@article{Flix2026,
  author = {Félix, Vanessa G. and Ostos, Rodolfo and Mena, Luis J. and Toral-Cruz, Homero and Ochoa-Brust, Alberto and Velarde-Alvarado, Pablo and González-Potes, Apolinar and Félix-Cuadras, Ramón A. and León-Borges, José A. and Martínez-Peláez, Rafael},
  title = {Quality Assessment of Generative AI in Cybersecurity Certification},
  journal = {Informatics},
  year = {2026},
  volume = {13},
  number = {4},
  pages = {53},
  doi = {10.3390/informatics13040053}
}
Kostenlos starten →