Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation
Frontiers in Digital Health · 18.11.2026 · 8
DOI: 10.3389/fdgth.2026.1981579 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 18.11.2026. Verfasst von Joe Hasei und 7 weiteren Autor:innen. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged.
Objective
We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates.
Methods
Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations.
Results
Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis.
Conclusions
Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.
Zitieren
APA
Hasei, J., Toda, Y., Nishimura, Y., Ikezaki, T., Matsuoka, H., Takeuchi, K., Nakahara, R., & Ozaki, T. (2026). Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation. Frontiers in Digital Health, 8. https://doi.org/10.3389/fdgth.2026.1981579
Vancouver
Hasei J, Toda Y, Nishimura Y, Ikezaki T, Matsuoka H, Takeuchi K, et al. Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation. Frontiers in Digital Health. 2026;8. doi:10.3389/fdgth.2026.1981579
BibTeX
@article{Hasei2026,
author = {Hasei, Joe and Toda, Yuichiro and Nishimura, Yuki and Ikezaki, Taichi and Matsuoka, Hiroki and Takeuchi, Koichi and Nakahara, Ryuichi and Ozaki, Toshifumi},
title = {Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation},
journal = {Frontiers in Digital Health},
year = {2026},
volume = {8},
doi = {10.3389/fdgth.2026.1981579}
}
EUVIMED