Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE
Applied Research Advances · 16.11.2026 · 1-21
DOI: 10.65069/ara31202741 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Research Advances, online erschienen am 16.11.2026. Verfasst von Temel Kaan Ekiz und 1 weiteren Autor:innen. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The requirement classification technique helps with planning, quality assurance, architecture decisions, and risk management. However, traditional metrics regard all misclassifications as equally costly. In this work, we evaluate Term Frequency-Inverse Document Frequency (TF-IDF) with Logistic Regression, TF-IDF with LinearSVC, Qwen3.5-4B, and Gemma 4 E2B IT on PROMISE_exp in two tasks: binary functional/non-functional classification and 11-class NFR subtype classification. In addition to common metrics, we provide four risk metrics: Risk-Weighted Error (RWE), Criticality Miss Rate (CMR) by triage level, Critical-Alert Precision (CAP), and Critical Subtype Error Rate (CSER). Uncertainty is measured with 2,000 paired class-stratified bootstrap replicates. LinearSVC achieves the highest accuracy both in Task A (0.907) and Task B (0.752), and the lowest Task B RWE (0.235). On gold-labelled NFRs, Qwen3.5-4B shows the best criticality-triage point estimates: CMR=0.073, CAP=0.911, and CSER=0.127. Its paired CMR and CSER differences against Gemma exclude zero, whereas its comparisons with the two classical baselines remain uncertain. Sensitivity analysis reveals that this advantage of Qwen3.5-4B does not persist when the six critical classes are considered equally important. Its differences against Gemma lose support when Gemma's five invalid outputs are removed. When Task A errors are included in triage, Qwen misses 19/55 critical requirements compared with 7/55 for LinearSVC. Thus, risk metrics give the most meaningful results in combination with class-specific, output validity, and pipeline-level analysis.
Zitieren
APA
Ekiz, T. K., & Önden, A. (2026). Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE. Applied Research Advances, 1-21. https://doi.org/10.65069/ara31202741
Vancouver
Ekiz TK, Önden A. Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE. Applied Research Advances. 2026:1-21. doi:10.65069/ara31202741
BibTeX
@article{Ekiz2026,
author = {Ekiz, Temel Kaan and Önden, Abdullah},
title = {Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE},
journal = {Applied Research Advances},
year = {2026},
pages = {1--21},
doi = {10.65069/ara31202741}
}
EUVIMED