EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE

Temel Kaan Ekiz, Abdullah Önden

Applied Research Advances · 16.11.2026 · 1-21

DOI: 10.65069/ara31202741 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Research Advances, online erschienen am 16.11.2026. Verfasst von Temel Kaan Ekiz und 1 weiteren Autor:innen. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The requirement classification technique helps with planning, quality assurance, architecture decisions, and risk management. However, traditional metrics regard all misclassifications as equally costly. In this work, we evaluate Term Frequency-Inverse Document Frequency (TF-IDF) with Logistic Regression, TF-IDF with LinearSVC, Qwen3.5-4B, and Gemma 4 E2B IT on PROMISE_exp in two tasks: binary functional/non-functional classification and 11-class NFR subtype classification. In addition to common metrics, we provide four risk metrics: Risk-Weighted Error (RWE), Criticality Miss Rate (CMR) by triage level, Critical-Alert Precision (CAP), and Critical Subtype Error Rate (CSER). Uncertainty is measured with 2,000 paired class-stratified bootstrap replicates. LinearSVC achieves the highest accuracy both in Task A (0.907) and Task B (0.752), and the lowest Task B RWE (0.235). On gold-labelled NFRs, Qwen3.5-4B shows the best criticality-triage point estimates: CMR=0.073, CAP=0.911, and CSER=0.127. Its paired CMR and CSER differences against Gemma exclude zero, whereas its comparisons with the two classical baselines remain uncertain. Sensitivity analysis reveals that this advantage of Qwen3.5-4B does not persist when the six critical classes are considered equally important. Its differences against Gemma lose support when Gemma's five invalid outputs are removed. When Task A errors are included in triage, Qwen misses 19/55 critical requirements compared with 7/55 for LinearSVC. Thus, risk metrics give the most meaningful results in combination with class-specific, output validity, and pipeline-level analysis.

Zitieren

APA

Ekiz, T. K., & Önden, A. (2026). Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE. Applied Research Advances, 1-21. https://doi.org/10.65069/ara31202741

Vancouver

Ekiz TK, Önden A. Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE. Applied Research Advances. 2026:1-21. doi:10.65069/ara31202741

BibTeX
@article{Ekiz2026,
  author = {Ekiz, Temel Kaan and Önden, Abdullah},
  title = {Risk-Aware Evaluation of Large Language Models for Requirements Classification: An Exploratory Study on PROMISE},
  journal = {Applied Research Advances},
  year = {2026},
  pages = {1--21},
  doi = {10.65069/ara31202741}
}
Kostenlos starten →