EUVIMEDEUVIMED entdecken
Crossref

Benchmarking Generative AI Models for Cybersecurity Vulnerability Identification Using 50 Curated Vulnerability Cases: An Exploratory Single-Pass Comparative Study of ChatGPT, Gemini, and Claude

Emanuel Dungon, Mark Jan Irog, Hans Luther Salva, Julse Lorenz Alvior, Rosa Mae Bascar, Rod Albert Aspera, Serafin Palmares, Kristine Soberano

International Journal of Transformative Multidisciplinary Studies · 2026-12 · 2 (4) · 42-64

DOI: 10.66074/r9l7p4j8w ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Transformative Multidisciplinary Studies, online erschienen am 05.10.2026. Verfasst von Emanuel Dungon und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

LLMs are being rapidly examined for cybersecurity applications, including identifying software vulnerabilities, conducting secure code reviews, and performing security analysis. Despite recent developments in Generative Artificial Intelligence (GenAI), there is a lack of empirical evidence on the relative performance of state-of-the-art commercial LLMs on discovering and categorizing software vulnerabilities. In this paper, we benchmarked the cybersecurity vulnerability-identification capabilities of ChatGPT, Gemini, and Claude on a dataset of fifty curated vulnerability cases from various Common Weakness Enumeration (CWE) categories. All the models were given the same prompts in a comparative benchmarking setup in a standardized testing environment. Responses were scored using a CWE-based scoring structure that includes exact matches, related matches based on established MITRE CWE links, and incorrect classifications. Descriptive statistics and the Friedman test were used to analyze performance. Descriptive analyses showed that ChatGPT obtained the highest weighted score (72/100), followed by Claude (66/100) and Gemini (64/100). ChatGPT also achieved the highest exact classification rate and the shortest observed response time under the experimental conditions. However, the inferential analysis indicated no statistically significant differences among the evaluated models (Friedman χ²(2) = 3.170, p = .205). These findings suggest broadly comparable vulnerability identification across the three LLMs under the exploratory, single-iteration benchmark conditions employed in the study. All evaluated models demonstrated meaningful capability for cybersecurity vulnerability identification. Although ChatGPT obtained the highest descriptive benchmark scores, inferential statistical analysis indicated no statistically significant performance differences among the evaluated models.

Zitieren

APA

Dungon, E., Irog, M. J., Salva, H. L., Alvior, J. L., Bascar, R. M., Aspera, R. A., Palmares, S., & Soberano, K. (2026). Benchmarking Generative AI Models for Cybersecurity Vulnerability Identification Using 50 Curated Vulnerability Cases: An Exploratory Single-Pass Comparative Study of ChatGPT, Gemini, and Claude. International Journal of Transformative Multidisciplinary Studies, 2(4), 42-64. https://doi.org/10.66074/r9l7p4j8w

Vancouver

Dungon E, Irog MJ, Salva HL, Alvior JL, Bascar RM, Aspera RA, et al. Benchmarking Generative AI Models for Cybersecurity Vulnerability Identification Using 50 Curated Vulnerability Cases: An Exploratory Single-Pass Comparative Study of ChatGPT, Gemini, and Claude. International Journal of Transformative Multidisciplinary Studies. 2026;2(4):42-64. doi:10.66074/r9l7p4j8w

BibTeX
@article{Dungon2026,
  author = {Dungon, Emanuel and Irog, Mark Jan and Salva, Hans Luther and Alvior, Julse Lorenz and Bascar, Rosa Mae and Aspera, Rod Albert and Palmares, Serafin and Soberano, Kristine},
  title = {Benchmarking Generative AI Models for Cybersecurity Vulnerability Identification Using 50 Curated Vulnerability Cases: An Exploratory Single-Pass Comparative Study of ChatGPT, Gemini, and Claude},
  journal = {International Journal of Transformative Multidisciplinary Studies},
  year = {2026},
  volume = {2},
  number = {4},
  pages = {42--64},
  doi = {10.66074/r9l7p4j8w}
}
Kostenlos starten →