EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems

Moon-Sup Lee, Seung-Yeon Han

Systems · 10.09.2026 · 14 (9) · 1126

DOI: 10.3390/systems14091126 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Systems, online erschienen am 10.09.2026. Verfasst von Moon-Sup Lee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Land, Infrastructure and Transport.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human-in-the-loop pipelines are widely used in artificial intelligence (AI)-based inspection systems, coupling an automated model, human reviewers, and an organizational reporting layer that drives decisions and, potentially, model retraining. The closed loop this creates is a design feature, not a defect; the pathology lies one level up, where a record produced by editing the model’s own output is treated as an independent reference and the automated-versus-reviewed difference is reported as “accuracy”—a closed loop read as open. This paper offers no alternative estimator; it shows that none is available here and that the infeasibility is itself the diagnosis. We examine the loop in a national pavement-management program, matching automated and reviewed outputs for 349 images across 162,520 individual grids. Reviewers accepted 89.06% of grids without change, added 10.25%, removed 0.28%, and reclassified 0.42% (an addition-to-removal ratio of 36.4:1); patching accounted for 64.5% of the added grids (39.8–69.0% across leave-one-route-out folds), added grids largely recording maintenance patches, not cracks. Consistent with the documented review workflow, 24 images were identical even by grid type and 98.06% of automated distress grids survived review (97.6–99.6% across folds); the reviewed record therefore cannot serve as an independent reference, and detection accuracy cannot be estimated. Across 40 route segments, the dependence mechanism and the asymmetry’s direction are invariant; the composition and magnitude are not. We reinterpret the reported figure and derive systems-level interventions—reporting items, interface priorities, and independent-reference requirements.

Förderung

  • Ministry of Land, Infrastructure and Transport

Zitieren

APA

Lee, M., & Han, S. (2026). When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems. Systems, 14(9), 1126. https://doi.org/10.3390/systems14091126

Vancouver

Lee M, Han S. When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems. Systems. 2026;14(9):1126. doi:10.3390/systems14091126

BibTeX
@article{Lee2026,
  author = {Lee, Moon-Sup and Han, Seung-Yeon},
  title = {When Human Review Is Treated as Ground Truth: A Socio-Technical Analysis of Reference Dependence and Measurement Validity in Human-in-the-Loop AI Inspection Systems},
  journal = {Systems},
  year = {2026},
  volume = {14},
  number = {9},
  pages = {1126},
  doi = {10.3390/systems14091126}
}
Kostenlos starten →