Evaluating GPT-5 and Fine-Tuned Transformer Models for QUS-Based User Story Quality Verification
Information · 28.10.2026 · 17 (11) · 1074
DOI: 10.3390/info17111074 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information, online erschienen am 28.10.2026. Verfasst von Leonardo Espinosa und 3 weiteren Autor:innen. Die Arbeit nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Secretaría de Ciencia, Humanidades, Tecnología e Innovación.
Abstract (Originalsprache)
User stories are widely used in agile requirements engineering, but their informal, natural language-based structure can introduce ambiguity, incomplete information, excessive scope, and other flaws that affect subsequent development activities. This study investigates the automated verification of individual Quality User Story (QUS) criteria by comparing general-purpose generative models evaluated using prompts with specialized Transformer models using supervised fine-tuning. Specifically, GPT-5 and GPT-5 Thinking were evaluated under Context-Minimal Prompting (CMP) and Context-Rich Prompting (CRP), while four task-specialized single-label (SL) models—SL-BUSQ-v2, SL-RUSQ-v2, SL-DUSQ-v2, and SL-gemma-USQ—were trained using a two-stage adaptation strategy. In the first stage, the models were pre-adapted on a partial multi-label (ML) corpus of 1625 user stories to expose them to the coexistence of multiple QUS violations. In the second stage, the models were adjusted for independent binary classification tasks on the eight individual QUS criteria using a refined reference set of 960 stories. The results show that all specialized models achieved higher average F1-scores than the GPT-5 prompting configurations. SL-gemma-USQ obtained the highest average F1-score (0.957793), followed by SL-BUSQ-v2 (0.956753), while the strongest prompting-based baseline, GPT-5 CMP, reached 0.915080. Overall, the findings suggest that supervised task specialization is a promising strategy for QUS-based user story quality verification.
Förderung
- Secretaría de Ciencia, Humanidades, Tecnología e Innovación
Zitieren
APA
Espinosa, L., Aguileta, A. A., Vera, R. A. A., & Flores-Rios, B. L. (2026). Evaluating GPT-5 and Fine-Tuned Transformer Models for QUS-Based User Story Quality Verification. Information, 17(11), 1074. https://doi.org/10.3390/info17111074
Vancouver
Espinosa L, Aguileta AA, Vera RAA, Flores-Rios BL. Evaluating GPT-5 and Fine-Tuned Transformer Models for QUS-Based User Story Quality Verification. Information. 2026;17(11):1074. doi:10.3390/info17111074
BibTeX
@article{Espinosa2026,
author = {Espinosa, Leonardo and Aguileta, Antonio A. and Vera, Raúl A. Aguilar and Flores-Rios, Brenda L.},
title = {Evaluating GPT-5 and Fine-Tuned Transformer Models for QUS-Based User Story Quality Verification},
journal = {Information},
year = {2026},
volume = {17},
number = {11},
pages = {1074},
doi = {10.3390/info17111074}
}
EUVIMED