EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Enhancing Software Code Vulnerability Detection Using GPT-4o and Claude-3.5 Sonnet: A Study on Prompt Engineering Techniques

Jaehyeon Bae, Seoryeong Kwon, Seunghwan Myeong

Electronics · 06.07.2024 · 13 (13) · 2657

DOI: 10.3390/electronics13132657 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 06.07.2024. Verfasst von Jaehyeon Bae und 2 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: The Ministry of Education of the Republic of Korea and the National Research Foundation of Korea.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study investigates the efficacy of advanced large language models, specifically GPT-4o, Claude-3.5 Sonnet, and GPT-3.5 Turbo, in detecting software vulnerabilities. Our experiment utilized vulnerable and secure code samples from the NIST Software Assurance Reference Dataset (SARD), focusing on C++, Java, and Python. We employed three distinct prompting techniques as follows: Concise, Tip Setting, and Step-by-Step. The results demonstrate that GPT-4o and Claude-3.5 Sonnet significantly outperform GPT-3.5 Turbo in vulnerability detection. GPT-4o showed the highest improvement with the Step-by-Step prompt, achieving an F1 score of 0.9072. Claude-3.5 Sonnet exhibited consistent high performance across all prompt types, with its Step-by-Step prompt yielding the best overall results (F1 score: 0.8933, AUC: 0.74). In contrast, GPT-3.5 Turbo showed minimal performance changes across prompts, with the Tip Setting prompt performing best (AUC: 0.65, F1 score: 0.6772), yet significantly lower than the other models. Our findings highlight the potential of advanced models in enhancing software security and underscore the importance of prompt engineering in optimizing their performance.

Förderung

  • The Ministry of Education of the Republic of Korea and the National Research Foundation of Korea

Zitieren

APA

Bae, J., Kwon, S., & Myeong, S. (2024). Enhancing Software Code Vulnerability Detection Using GPT-4o and Claude-3.5 Sonnet: A Study on Prompt Engineering Techniques. Electronics, 13(13), 2657. https://doi.org/10.3390/electronics13132657

Vancouver

Bae J, Kwon S, Myeong S. Enhancing Software Code Vulnerability Detection Using GPT-4o and Claude-3.5 Sonnet: A Study on Prompt Engineering Techniques. Electronics. 2024;13(13):2657. doi:10.3390/electronics13132657

BibTeX
@article{Bae2024,
  author = {Bae, Jaehyeon and Kwon, Seoryeong and Myeong, Seunghwan},
  title = {Enhancing Software Code Vulnerability Detection Using GPT-4o and Claude-3.5 Sonnet: A Study on Prompt Engineering Techniques},
  journal = {Electronics},
  year = {2024},
  volume = {13},
  number = {13},
  pages = {2657},
  doi = {10.3390/electronics13132657}
}
Kostenlos starten →