CySecBERT : A Domain-Adapted Language Model for the Cybersecurity Domain
ACM Transactions on Privacy and Security · 31.05.2024 · 27 (2) · 1-20
DOI: 10.1145/3652594 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Privacy and Security, online erschienen am 08.04.2024. Verfasst von Markus Bayer und 3 weiteren Autor:innen. Die Arbeit wurde bisher 76-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts, National Research Center for Applied Cybersecurity ATHENE, German Federal Ministry for Education and Research.
Abstract (Originalsprache)
The field of cysec is evolving fast. Security professionals are in need of intelligence on past, current and —ideally — upcoming threats, because attacks are becoming more advanced and are increasingly targeting larger and more complex systems. Since the processing and analysis of such large amounts of information cannot be addressed manually, cysec experts rely on machine learning techniques. In the textual domain, pre-trained language models such as Bidirectional Encoder Representations from Transformers (BERT) have proven to be helpful as they provide a good baseline for further fine-tuning. However, due to the domain-knowledge and the many technical terms in cysec, general language models might miss the gist of textual information. For this reason, we create a high-quality dataset 1 and present a language model 2 specifically tailored to the cysec domain that can serve as a basic building block for cybersecurity systems. The model is compared on 15 tasks: Domain-dependent extrinsic tasks for measuring the performance on specific problems, intrinsic tasks for measuring the performance of the internal representations of the model, as well as general tasks from the SuperGLUE benchmark. The results of the intrinsic tasks show that our model improves the internal representation space of domain words compared with the other models. The extrinsic, domain-dependent tasks, consisting of sequence tagging and classification, show that the model performs best in cybersecurity scenarios. In addition, we pay special attention to the choice of hyperparameters against catastrophic forgetting, as pre-trained models tend to forget the original knowledge during further training.
Förderung
- German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts
- National Research Center for Applied Cybersecurity ATHENE
- German Federal Ministry for Education and Research
- CYLENCE
Zitieren
APA
Bayer, M., Kuehn, P., Shanehsaz, R., & Reuter, C. (2024). CySecBERT : A Domain-Adapted Language Model for the Cybersecurity Domain. ACM Transactions on Privacy and Security, 27(2), 1-20. https://doi.org/10.1145/3652594
Vancouver
Bayer M, Kuehn P, Shanehsaz R, Reuter C. CySecBERT : A Domain-Adapted Language Model for the Cybersecurity Domain. ACM Transactions on Privacy and Security. 2024;27(2):1-20. doi:10.1145/3652594
BibTeX
@article{Bayer2024,
author = {Bayer, Markus and Kuehn, Philipp and Shanehsaz, Ramin and Reuter, Christian},
title = {CySecBERT : A Domain-Adapted Language Model for the Cybersecurity Domain},
journal = {ACM Transactions on Privacy and Security},
year = {2024},
volume = {27},
number = {2},
pages = {1--20},
doi = {10.1145/3652594}
}
EUVIMED