EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models

Fouad Trad, Ali Chehab

Machine Learning and Knowledge Extraction · 2024-02-06 · 6 (1) · 367-384

DOI: 10.3390/make6010018 ↗

At a glance

Journal article in Machine Learning and Knowledge Extraction, published online on 2024-02-06. Written by Fouad Trad and 1 co-authors. Cited 122 times so far (Crossref); 64 references. The full text is freely available (open access, CC BY 4.0). Funded by: Maroun Semaan Faculty of Engineering and Architecture (MSFEA) at the American University of Beirut (AUB).

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Large Language Models (LLMs) are reshaping the landscape of Machine Learning (ML) application development. The emergence of versatile LLMs capable of undertaking a wide array of tasks has reduced the necessity for intensive human involvement in training and maintaining ML models. Despite these advancements, a pivotal question emerges: can these generalized models negate the need for task-specific models? This study addresses this question by comparing the effectiveness of LLMs in detecting phishing URLs when utilized with prompt-engineering techniques versus when fine-tuned. Notably, we explore multiple prompt-engineering strategies for phishing URL detection and apply them to two chat models, GPT-3.5-turbo and Claude 2. In this context, the maximum result achieved was an F1-score of 92.74% by using a test set of 1000 samples. Following this, we fine-tune a range of base LLMs, including GPT-2, Bloom, Baby LLaMA, and DistilGPT-2—all primarily developed for text generation—exclusively for phishing URL detection. The fine-tuning approach culminated in a peak performance, achieving an F1-score of 97.29% and an AUC of 99.56% on the same test set, thereby outperforming existing state-of-the-art methods. These results highlight that while LLMs harnessed through prompt engineering can expedite application development processes, achieving a decent performance, they are not as effective as dedicated, task-specific LLMs.

Funding

  • Maroun Semaan Faculty of Engineering and Architecture (MSFEA) at the American University of Beirut (AUB)

Cite

APA

Trad, F., & Chehab, A. (2024). Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models. Machine Learning and Knowledge Extraction, 6(1), 367-384. https://doi.org/10.3390/make6010018

Vancouver

Trad F, Chehab A. Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models. Machine Learning and Knowledge Extraction. 2024;6(1):367-384. doi:10.3390/make6010018

BibTeX
@article{Trad2024,
  author = {Trad, Fouad and Chehab, Ali},
  title = {Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2024},
  volume = {6},
  number = {1},
  pages = {367--384},
  doi = {10.3390/make6010018}
}
Start for free →