Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models
Machine Learning and Knowledge Extraction · 06.02.2024 · 6 (1) · 367-384
DOI: 10.3390/make6010018 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machine Learning and Knowledge Extraction, online erschienen am 06.02.2024. Verfasst von Fouad Trad und 1 weiteren Autor:innen. Die Arbeit wurde bisher 122-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Maroun Semaan Faculty of Engineering and Architecture (MSFEA) at the American University of Beirut (AUB).
Abstract (Originalsprache)
Large Language Models (LLMs) are reshaping the landscape of Machine Learning (ML) application development. The emergence of versatile LLMs capable of undertaking a wide array of tasks has reduced the necessity for intensive human involvement in training and maintaining ML models. Despite these advancements, a pivotal question emerges: can these generalized models negate the need for task-specific models? This study addresses this question by comparing the effectiveness of LLMs in detecting phishing URLs when utilized with prompt-engineering techniques versus when fine-tuned. Notably, we explore multiple prompt-engineering strategies for phishing URL detection and apply them to two chat models, GPT-3.5-turbo and Claude 2. In this context, the maximum result achieved was an F1-score of 92.74% by using a test set of 1000 samples. Following this, we fine-tune a range of base LLMs, including GPT-2, Bloom, Baby LLaMA, and DistilGPT-2—all primarily developed for text generation—exclusively for phishing URL detection. The fine-tuning approach culminated in a peak performance, achieving an F1-score of 97.29% and an AUC of 99.56% on the same test set, thereby outperforming existing state-of-the-art methods. These results highlight that while LLMs harnessed through prompt engineering can expedite application development processes, achieving a decent performance, they are not as effective as dedicated, task-specific LLMs.
Förderung
- Maroun Semaan Faculty of Engineering and Architecture (MSFEA) at the American University of Beirut (AUB)
Zitieren
APA
Trad, F., & Chehab, A. (2024). Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models. Machine Learning and Knowledge Extraction, 6(1), 367-384. https://doi.org/10.3390/make6010018
Vancouver
Trad F, Chehab A. Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models. Machine Learning and Knowledge Extraction. 2024;6(1):367-384. doi:10.3390/make6010018
BibTeX
@article{Trad2024,
author = {Trad, Fouad and Chehab, Ali},
title = {Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models},
journal = {Machine Learning and Knowledge Extraction},
year = {2024},
volume = {6},
number = {1},
pages = {367--384},
doi = {10.3390/make6010018}
}
EUVIMED