Detection of Malicious Codes Generated by Large Language Models: A Comparison of GPT-3.5, GPT-4o, Gemini, and Claude
International Journal of Information Security Science · 2025
DOI: 10.55859/ijiss.1634763 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Information Security Science, 2025. Verfasst von Meltem Kurt Pehlivanoğlu und 1 weiteren Autor:innen. Die Arbeit nennt 17 Literaturangaben.
Abstract (Originalsprache)
This study presents novel machine learning-based approaches for detecting whether source code generated by Large Language Models (LLMs) contains malicious code. To achieve this, comprehensive datasets comprising malicious and benign code samples were created using the GPT-3.5 (ChatGPT), GPT-4o, Gemini, and Claude language models. The extracted code samples were then processed through CodeBERT, CodeT5, and manual feature extraction techniques before being classified using various machine learning algorithms. Experimental results demonstrate that this approach can effectively detect malicious software in code generated by LLMs. This study makes contributions to software security and represents a crucial step toward preventing the misuse of LLMs for malicious purposes. Moreover, the Random Forest algorithm for binary malicious code classification in LLM-generated code achieved the best F$_{1}$ score of 94.92\% on the ChatGPT-generated dataset (with CodeT5 feature extraction technique). We also showed that the classification models exhibited poor performance on the dataset generated by Claude language model.
Zitieren
APA
Pehlivanoğlu, M. K., & Çoban, M. G. (2025). Detection of Malicious Codes Generated by Large Language Models: A Comparison of GPT-3.5, GPT-4o, Gemini, and Claude. International Journal of Information Security Science. https://doi.org/10.55859/ijiss.1634763
Vancouver
Pehlivanoğlu MK, Çoban MG. Detection of Malicious Codes Generated by Large Language Models: A Comparison of GPT-3.5, GPT-4o, Gemini, and Claude. International Journal of Information Security Science. 2025. doi:10.55859/ijiss.1634763
BibTeX
@article{Pehlivanolu2025,
author = {Pehlivanoğlu, Meltem Kurt and Çoban, Murat Görkem},
title = {Detection of Malicious Codes Generated by Large Language Models: A Comparison of GPT-3.5, GPT-4o, Gemini, and Claude},
journal = {International Journal of Information Security Science},
year = {2025},
doi = {10.55859/ijiss.1634763}
}
EUVIMED