EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation

Joe Hasei, Yuichiro Toda, Yuki Nishimura, Taichi Ikezaki, Hiroki Matsuoka, Koichi Takeuchi, Ryuichi Nakahara, Toshifumi Ozaki

Frontiers in Digital Health · 18.11.2026 · 8

DOI: 10.3389/fdgth.2026.1981579 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 18.11.2026. Verfasst von Joe Hasei und 7 weiteren Autor:innen. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged.

Objective

We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates.

Methods

Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations.

Results

Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis.

Conclusions

Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.

Zitieren

APA

Hasei, J., Toda, Y., Nishimura, Y., Ikezaki, T., Matsuoka, H., Takeuchi, K., Nakahara, R., & Ozaki, T. (2026). Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation. Frontiers in Digital Health, 8. https://doi.org/10.3389/fdgth.2026.1981579

Vancouver

Hasei J, Toda Y, Nishimura Y, Ikezaki T, Matsuoka H, Takeuchi K, et al. Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation. Frontiers in Digital Health. 2026;8. doi:10.3389/fdgth.2026.1981579

BibTeX
@article{Hasei2026,
  author = {Hasei, Joe and Toda, Yuichiro and Nishimura, Yuki and Ikezaki, Taichi and Matsuoka, Hiroki and Takeuchi, Koichi and Nakahara, Ryuichi and Ozaki, Toshifumi},
  title = {Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation},
  journal = {Frontiers in Digital Health},
  year = {2026},
  volume = {8},
  doi = {10.3389/fdgth.2026.1981579}
}
Kostenlos starten →