Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification
Computers · 2026-07-07 · 15 (7) · 430
DOI: 10.3390/computers15070430 ↗At a glance
Journal article in Computers, published online on 2026-07-07. Written by Yahya Masri and 4 co-authors. Cited 1 times so far (Crossref); 99 references. The full text is freely available (open access, CC BY 4.0). Funded by: NSF I/UCRC Program, NASA AIST Program, NASA Goddard CISTO.
Abstract (original language)
System logs are essential for monitoring and diagnosing modern computing infrastructure; however, their scale and complexity require reliable automated interpretation. Although severity levels are predefined metadata, treating classification as an end task provides limited insight into a model’s ability to understand logs. Instead, severity classification can serve as a benchmark for probing runtime log comprehension. Using real-world journalctl data from Linux production servers, this study evaluates nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal clear performance stratification. Qwen3-4B achieves the highest accuracy (95.6%) with RAG, while Gemma3-1B improves substantially from 20.25% to 85.28%, and Qwen3-0.6B reaches 88.12% despite weak baseline performance. In contrast, several SRLMs exhibit performance degradation when paired with RAG. Efficiency further differentiates the models. Most Gemma and Llama variants complete inference in under 1.2 s per log, whereas Phi-4-Mini-Reasoning requires more than 228 s while achieving less than 10% accuracy. These findings indicate that architectural design, training objectives, and the ability to integrate retrieved context jointly determine performance. Overall, the results support severity classification as a practical lens for evaluating model competence and real-time deployability in digital twin systems.
Funding
- NSF I/UCRC Program
- NASA AIST Program
- NASA Goddard CISTO
- ASSIP program
- George Mason University’s College of Science
Cite
APA
Masri, Y., Ma, E., Wang, Z., Rogers, J., & Yang, C. (2026). Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification. Computers, 15(7), 430. https://doi.org/10.3390/computers15070430
Vancouver
Masri Y, Ma E, Wang Z, Rogers J, Yang C. Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification. Computers. 2026;15(7):430. doi:10.3390/computers15070430
BibTeX
@article{Masri2026,
author = {Masri, Yahya and Ma, Emily and Wang, Zifu and Rogers, Joseph and Yang, Chaowei},
title = {Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification},
journal = {Computers},
year = {2026},
volume = {15},
number = {7},
pages = {430},
doi = {10.3390/computers15070430}
}
EUVIMED