EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models

Aaron Conway, Adriana Hada, Jessica Schluter, Hui Grace Xu, Dan Lowden, Tim Miller, Ken Donald, Andrew Teodorczuk

JMIR Nursing · 05.10.2026 · 9 · e106133-e106133

DOI: 10.2196/106133 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Nursing, online erschienen am 05.10.2026. Verfasst von Aaron Conway und 7 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Healthcare Transformation, Queensland University of Technology, Park Rd, Kelvin Grove, Brisbane, QLD, 4059, Australia, 61 731383887. Hauptthemen laut MEDLINE: Patient Handoff, Artificial Intelligence. Die Arbeit nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical handover is the process during which responsibility and accountability for care are transferred between clinicians. AI has the potential to improve the reliability and completeness of clinical handover by helping clinicians detect predefined content areas that have been communicated, identify explicit information gaps, and prompt clarification before responsibility is transferred.

Objective

This study evaluated the performance of several large language models and prompt optimization strategies for structured information extraction of synthetic nursing handover transcripts.

Methods

Two registered nurses independently annotated a dataset of 203 synthetic handover transcripts to produce consensus labels for information extraction tasks. Tasks included (1) labeling spans of text into SBAR (Situation, Background, Assessment, Recommendation) categories, (2) content detection to determine if specific pieces of information were communicated, and (3) labeling spans of text that communicated information using uncertain terms that included a subtask for identifying unknown facts. Baseline and Genetic-Pareto (GEPA)–optimized prompts were compared for the GPT-5.2, GPT-5-nano, and MedGemma 27B large language models. Additionally, the LangExtract framework was evaluated for span-extraction tasks.

Results

The GPT-5.2–optimized model achieved a micro –F 1 -score of 0.85 (95% CI 0.83‐0.88) for content detection, an absolute improvement of +0.08 compared with the matched baseline. GPT-5-nano also performed better after optimization for content detection (micro –F 1 -score 0.81, 95% CI 0.78‐0.84), suggesting that this structured task was not limited to the highest-capacity model. For SBAR span extraction, GPT-5.2 with prompt optimization achieved a micro –F 1 -score of 0.76 (95% CI 0.72‐0.79), improving by +0.24 compared with baseline and exceeding LangExtract; GPT-5-nano also improved to a micro –F 1 -score of 0.69 (95% CI 0.66‐0.72). Broad uncertainty-span extraction remained comparatively weak despite prompt optimization (micro –F 1 -score 0.41, 95% CI 0.33‐0.48; absolute improvement +0.06). In contrast, explicit unknown-fact extraction was more accurate with GPT-5.2 (micro –F 1 -score 0.84, 95% CI 0.63‐1.00), GPT-5-nano (micro –F 1 -score, 0.84 95% CI 0.63‐1.00), and MedGemma 27B (micro –F 1 -score 0.80, 95% CI 0.63‐1.00). Genetic-Pareto–optimized prompts outperformed the LangExtract approach across each span-extraction task.

Conclusions

Prompt optimization improved matched-model point estimates, with the highest performance observed for predefined content detection and SBAR span extraction. Broad uncertainty extraction remained less accurate than the narrower unknown-fact task. These technical results do not establish clinical effectiveness, safety, or readiness for real-time use. Validation using authentic nursing handover communication and prospective evaluation in clinical workflows are required before clinical application.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Patient HandoffArtificial IntelligenceLarge Language ModelsHumans

Stichwörter der Autor:innen

artificial intelligenceclinical handoverinformation extractionlarge language modelsmachine learningnatural language processingnursingpatient handoffpatient safetyprompt engineering

Autor:innen und Institutionen

  • Centre for Healthcare Transformation, Queensland University of Technology, Park Rd, Kelvin Grove, Brisbane, QLD, 4059, Australia, 61 731383887.
  • School of Nursing, Queensland University of Technology, Brisbane, Australia.
  • The Prince Charles Hospital, Metro North Health, Brisbane, Australia.
  • Caboolture Hospital, Brisbane, Australia.
  • School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Australia.
  • School of Medicine & Dentistry, Griffith University, Gold Coast, Queensland, Australia.

Zitieren

APA

Conway, A., Hada, A., Schluter, J., Xu, H. G., Lowden, D., Miller, T., Donald, K., & Teodorczuk, A. (2026). AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models. JMIR Nursing, 9, e106133-e106133. https://doi.org/10.2196/106133

Vancouver

Conway A, Hada A, Schluter J, Xu HG, Lowden D, Miller T, et al. AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models. JMIR Nursing. 2026;9:e106133-e106133. doi:10.2196/106133

BibTeX
@article{Conway2026,
  author = {Conway, Aaron and Hada, Adriana and Schluter, Jessica and Xu, Hui Grace and Lowden, Dan and Miller, Tim and Donald, Ken and Teodorczuk, Andrew},
  title = {AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models},
  journal = {JMIR Nursing},
  year = {2026},
  volume = {9},
  pages = {e106133--e106133},
  doi = {10.2196/106133}
}
Kostenlos starten →