Cross-model evaluation of phishing detectors against LLM-generated emails
Frontiers in Big Data · 07.07.2026 · 9
DOI: 10.3389/fdata.2026.1883452 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Big Data, online erschienen am 07.07.2026. Verfasst von Rommel Gutierrez und 2 weiteren Autor:innen. Beteiligte Einrichtung: Escuela de Ingeniería en Ciberseguridad, FICA, Universidad de las Américas, Quito, Ecuador. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Phishing remains a prevalent cyberattack vector, and the widespread adoption of large language models (LLMs) has enabled adversaries to generate grammatically correct and contextually coherent phishing emails at scale, against which conventional detection systems are less effective. Although stylometric methods achieve over 95% accuracy within a single generator, their performance has not been systematically evaluated when the source model changes between training and deployment. This represents a significant gap, as adversaries can switch generators rapidly. A balanced corpus of 9,986 phishing emails was assembled, comprising 4,986 emails generated by three modern LLMs (GPT-4.1, DeepSeek 3.2, and Llama 3.3 70B) across five thematic categories, and 5,000 human phishing emails sampled in a stratified manner from five public sources. Seventeen stylometric features were extracted, and Logistic Regression and XGBoost classifiers were evaluated under intra-model, cross-model, threshold-recalibrated, cross-dataset, and aggregated-pool settings. Intra-model F1 scores reached 0.96 under stratified cross-validation and 0.999 on held-out splits used for the cross-model matrix. However, cross-model F1 dropped by 28.0 percentage points under the default decision threshold of 0.5. Notably, the area under the receiver operating characteristic curve remained above 0.96 in every off-diagonal cell, indicating that discriminative information is preserved even though the decision threshold is generator-specific. Recalibrating the threshold on a small target subset reduced the gap to 4.0 percentage points (an 86% reduction), and an aggregated-pool detector achieved F1 = 0.997 on each generator. This work reframes cross-model phishing detection from a problem of model incompatibility to one of practical calibration, and provides two deployable solutions, threshold recalibration on a small target slice and aggregated-pool training, along with a publicly released multi-LLM corpus.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Escuela de Ingeniería en Ciberseguridad, FICA, Universidad de las Américas, Quito, Ecuador.
Zitieren
APA
Gutierrez, R., Villegas-Ch, W., & Govea, J. (2026). Cross-model evaluation of phishing detectors against LLM-generated emails. Frontiers in Big Data, 9. https://doi.org/10.3389/fdata.2026.1883452
Vancouver
Gutierrez R, Villegas-Ch W, Govea J. Cross-model evaluation of phishing detectors against LLM-generated emails. Frontiers in Big Data. 2026;9. doi:10.3389/fdata.2026.1883452
BibTeX
@article{Gutierrez2026,
author = {Gutierrez, Rommel and Villegas-Ch, William and Govea, Jaime},
title = {Cross-model evaluation of phishing detectors against LLM-generated emails},
journal = {Frontiers in Big Data},
year = {2026},
volume = {9},
doi = {10.3389/fdata.2026.1883452}
}
EUVIMED