Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot
Applied Sciences · 20.11.2026 · 16 (22) · 11506
DOI: 10.3390/app162211506 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.11.2026. Verfasst von Mustafa Kacan und 1 weiteren Autor:innen. Die Arbeit nennt 76 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This study asks whether the Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) cycle can improve utterance capture in a live enterprise chatbot, and how much of the change is attributable to the intervention during a concurrent migration to large language model (LLM) components. Turkish WhatsApp utterances of Turkish Airlines’ TK Assistant (80,295 utterances, January–July 2025) was analyzed; utterance capture scores (success threshold 0.95) were graded on a five-level scale, with 0.95 as the success threshold. After bidirectional Pareto analysis, 349 Strong-band utterances without a verbatim counterpart were added to ten sub-intent libraries. Assessment used utterance-level re-capture analysis against non-added controls, expected-versus-actual attribution, Laney p′ charts and blind human validation of 300 utterances. Recurring added texts were captured as exact matches (17 of 22 at first recurrence, against none of eight recurring controls; difference 77 points, 95% confidence interval 39–90), without organic promotion beyond near-identical forms. Under stated assumptions, the intervention was followed by 69 additional exact matches, while the successful utterance capture rate fell from 21.04% to 16.11%, a decline inseparable from the migration. Annotators judged exact matches correctly routed no more often than Strong-band matches (87% vs. 98%, p = 0.032 jointly, marginal per annotator). Verbatim curation buys exact-match efficiency, not understanding; confidence-based metrics should be secondary improvement targets.
Zitieren
APA
Kacan, M., & Alp, S. (2026). Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot. Applied Sciences, 16(22), 11506. https://doi.org/10.3390/app162211506
Vancouver
Kacan M, Alp S. Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot. Applied Sciences. 2026;16(22):11506. doi:10.3390/app162211506
BibTeX
@article{Kacan2026,
author = {Kacan, Mustafa and Alp, Selcuk},
title = {Efficiency Without Quality? A Six Sigma DMAIC (Define, Measure, Analyze, Improve, Control) Field Study of Sub-Intent Library Curation in a Live Airline Chatbot},
journal = {Applied Sciences},
year = {2026},
volume = {16},
number = {22},
pages = {11506},
doi = {10.3390/app162211506}
}
EUVIMED