EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification

Yahya Masri, Emily Ma, Zifu Wang, Joseph Rogers, Chaowei Yang

Computers · 07.07.2026 · 15 (7) · 430

DOI: 10.3390/computers15070430 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 07.07.2026. Verfasst von Yahya Masri und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 99 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSF I/UCRC Program, NASA AIST Program, NASA Goddard CISTO.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

System logs are essential for monitoring and diagnosing modern computing infrastructure; however, their scale and complexity require reliable automated interpretation. Although severity levels are predefined metadata, treating classification as an end task provides limited insight into a model’s ability to understand logs. Instead, severity classification can serve as a benchmark for probing runtime log comprehension. Using real-world journalctl data from Linux production servers, this study evaluates nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal clear performance stratification. Qwen3-4B achieves the highest accuracy (95.6%) with RAG, while Gemma3-1B improves substantially from 20.25% to 85.28%, and Qwen3-0.6B reaches 88.12% despite weak baseline performance. In contrast, several SRLMs exhibit performance degradation when paired with RAG. Efficiency further differentiates the models. Most Gemma and Llama variants complete inference in under 1.2 s per log, whereas Phi-4-Mini-Reasoning requires more than 228 s while achieving less than 10% accuracy. These findings indicate that architectural design, training objectives, and the ability to integrate retrieved context jointly determine performance. Overall, the results support severity classification as a practical lens for evaluating model competence and real-time deployability in digital twin systems.

Förderung

  • NSF I/UCRC Program
  • NASA AIST Program
  • NASA Goddard CISTO
  • ASSIP program
  • George Mason University’s College of Science

Zitieren

APA

Masri, Y., Ma, E., Wang, Z., Rogers, J., & Yang, C. (2026). Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification. Computers, 15(7), 430. https://doi.org/10.3390/computers15070430

Vancouver

Masri Y, Ma E, Wang Z, Rogers J, Yang C. Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification. Computers. 2026;15(7):430. doi:10.3390/computers15070430

BibTeX
@article{Masri2026,
  author = {Masri, Yahya and Ma, Emily and Wang, Zifu and Rogers, Joseph and Yang, Chaowei},
  title = {Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {7},
  pages = {430},
  doi = {10.3390/computers15070430}
}
Kostenlos starten →