When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems
Systems · 10.09.2026 · 14 (9) · 1126
DOI: 10.3390/systems14091126 ↗Kurz gefasst
Fachartikel in der Zeitschrift Systems, online erschienen am 10.09.2026. Verfasst von Moon-Sup Lee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Land, Infrastructure and Transport.
Abstract (Originalsprache)
Human-in-the-loop pipelines are widely used in artificial intelligence (AI)-based inspection systems, coupling an automated model, human reviewers, and an organizational reporting layer that drives decisions and, potentially, model retraining. The closed loop this creates is a design feature, not a defect; the pathology lies one level up, where a record produced by editing the model’s own output is treated as an independent reference and the automated-versus-reviewed difference is reported as “accuracy”—a closed loop read as open. This paper offers no alternative estimator; it shows that none is available here and that the infeasibility is itself the diagnosis. We examine the loop in a national pavement-management program, matching automated and reviewed outputs for 349 images across 162,520 individual grids. Reviewers accepted 89.06% of grids without change, added 10.25%, removed 0.28%, and reclassified 0.42% (an addition-to-removal ratio of 36.4:1); patching accounted for 64.5% of the added grids (39.8–69.0% across leave-one-route-out folds), added grids largely recording maintenance patches, not cracks. Consistent with the documented review workflow, 24 images were identical even by grid type and 98.06% of automated distress grids survived review (97.6–99.6% across folds); the reviewed record therefore cannot serve as an independent reference, and detection accuracy cannot be estimated. Across 40 route segments, the dependence mechanism and the asymmetry’s direction are invariant; the composition and magnitude are not. We reinterpret the reported figure and derive systems-level interventions—reporting items, interface priorities, and independent-reference requirements.
Förderung
- Ministry of Land, Infrastructure and Transport
Zitieren
APA
Lee, M., & Han, S. (2026). When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems. Systems, 14(9), 1126. https://doi.org/10.3390/systems14091126
Vancouver
Lee M, Han S. When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems. Systems. 2026;14(9):1126. doi:10.3390/systems14091126
BibTeX
@article{Lee2026,
author = {Lee, Moon-Sup and Han, Seung-Yeon},
title = {When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems},
journal = {Systems},
year = {2026},
volume = {14},
number = {9},
pages = {1126},
doi = {10.3390/systems14091126}
}
EUVIMED