EUVIMEDObjevte EUVIMED
VergleichsstudieOpen access

AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models

KI-basierte strukturierte Informationsextraktion aus synthetischen Pflegeübergabe-Transkripten: vergleichende Bewertung großer Sprachmodelle

Aaron Conway, Adriana Hada, Jessica Schluter, Hui Grace Xu, Dan Lowden, Tim Miller, Ken Donald, Andrew Teodorczuk

JMIR Nursing · 2026-10-05 · 9 · e106133-e106133

DOI: 10.2196/106133 ↗

Auf einen Blick

Große Sprachmodelle extrahierten aus 203 synthetischen Pflegeübergaben strukturierte Informationen teils zuverlässig. Die beste Leistung zeigte sich bei der Detektion vorgegebener Inhalte und bei SBAR-Span-Extraktion.

Fragestellung

Wie gut erkennen verschiedene große Sprachmodelle strukturierte Informationen in synthetischen Pflegeübergabe-Transkripten, und verbessert Prompt-Optimierung die Leistung?

Methode

Zwei diplomierte Pflegefachpersonen annotierten 203 synthetische Transkripte. Verglichen wurden Baseline- und GEPA-optimierte Prompts für GPT-5.2, GPT-5-nano und MedGemma 27B; LangExtract wurde für Span-Extraktion geprüft.

Ergebnisse

GPT-5.2 optimiert: Content Detection micro-F1 0,85 (95 %-KI 0,83–0,88), +0,08 gegenüber Baseline. SBAR-Extraktion: 0,76 (95 %-KI 0,72–0,79), +0,24. Broad uncertainty: 0,41 (95 %-KI 0,33–0,48).

Einschränkungen

Die technischen Ergebnisse belegen keine klinische Wirksamkeit, Sicherheit oder Einsatzbereitschaft für den Echtzeiteinsatz. Validierung mit authentischer Pflegeübergabe und prospektive Evaluation im klinischen Ablauf fehlen.

Schlussfolgerung der Autor:innen

Prompt-Optimierung verbesserte die Leistung der verglichenen Modelle. Am besten schnitten die vorgegebene Inhaltsdetektion und die SBAR-Span-Extraktion ab; breite Unsicherheits-Extraktion blieb ungenauer.

Kernaussagen

  • Content Detection: GPT-5.2 optimiert micro-F1 0,85 (95 %-KI 0,83–0,88)
  • SBAR-Span-Extraktion: GPT-5.2 optimiert micro-F1 0,76 (95 %-KI 0,72–0,79)
  • Broad uncertainty blieb schwächer als die Extraktion unbekannter Fakten

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Vergleichsstudie in JMIR Nursing, published online on 2026-10-05. Written by Aaron Conway and 7 co-authors. Institution: Centre for Healthcare Transformation, Queensland University of Technology, Park Rd, Kelvin Grove, Brisbane, QLD, 4059, Australia, 61 731383887. Major MEDLINE topics: Patient Handoff, Artificial Intelligence. 46 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Clinical handover is the process during which responsibility and accountability for care are transferred between clinicians. AI has the potential to improve the reliability and completeness of clinical handover by helping clinicians detect predefined content areas that have been communicated, identify explicit information gaps, and prompt clarification before responsibility is transferred.

Objective

This study evaluated the performance of several large language models and prompt optimization strategies for structured information extraction of synthetic nursing handover transcripts.

Methods

Two registered nurses independently annotated a dataset of 203 synthetic handover transcripts to produce consensus labels for information extraction tasks. Tasks included (1) labeling spans of text into SBAR (Situation, Background, Assessment, Recommendation) categories, (2) content detection to determine if specific pieces of information were communicated, and (3) labeling spans of text that communicated information using uncertain terms that included a subtask for identifying unknown facts. Baseline and Genetic-Pareto (GEPA)–optimized prompts were compared for the GPT-5.2, GPT-5-nano, and MedGemma 27B large language models. Additionally, the LangExtract framework was evaluated for span-extraction tasks.

Results

The GPT-5.2–optimized model achieved a micro –F 1 -score of 0.85 (95% CI 0.83‐0.88) for content detection, an absolute improvement of +0.08 compared with the matched baseline. GPT-5-nano also performed better after optimization for content detection (micro –F 1 -score 0.81, 95% CI 0.78‐0.84), suggesting that this structured task was not limited to the highest-capacity model. For SBAR span extraction, GPT-5.2 with prompt optimization achieved a micro –F 1 -score of 0.76 (95% CI 0.72‐0.79), improving by +0.24 compared with baseline and exceeding LangExtract; GPT-5-nano also improved to a micro –F 1 -score of 0.69 (95% CI 0.66‐0.72). Broad uncertainty-span extraction remained comparatively weak despite prompt optimization (micro –F 1 -score 0.41, 95% CI 0.33‐0.48; absolute improvement +0.06). In contrast, explicit unknown-fact extraction was more accurate with GPT-5.2 (micro –F 1 -score 0.84, 95% CI 0.63‐1.00), GPT-5-nano (micro –F 1 -score, 0.84 95% CI 0.63‐1.00), and MedGemma 27B (micro –F 1 -score 0.80, 95% CI 0.63‐1.00). Genetic-Pareto–optimized prompts outperformed the LangExtract approach across each span-extraction task.

Conclusions

Prompt optimization improved matched-model point estimates, with the highest performance observed for predefined content detection and SBAR span extraction. Broad uncertainty extraction remained less accurate than the narrower unknown-fact task. These technical results do not establish clinical effectiveness, safety, or readiness for real-time use. Validation using authentic nursing handover communication and prospective evaluation in clinical workflows are required before clinical application.

Subjects

MeSH terms (MEDLINE)

Patient HandoffArtificial IntelligenceLarge Language ModelsHumans

Author keywords

artificial intelligenceclinical handoverinformation extractionlarge language modelsmachine learningnatural language processingnursingpatient handoffpatient safetyprompt engineering

Authors and institutions

  • Centre for Healthcare Transformation, Queensland University of Technology, Park Rd, Kelvin Grove, Brisbane, QLD, 4059, Australia, 61 731383887.
  • School of Nursing, Queensland University of Technology, Brisbane, Australia.
  • The Prince Charles Hospital, Metro North Health, Brisbane, Australia.
  • Caboolture Hospital, Brisbane, Australia.
  • School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, Australia.
  • School of Medicine & Dentistry, Griffith University, Gold Coast, Queensland, Australia.

Cite

APA

Conway, A., Hada, A., Schluter, J., Xu, H. G., Lowden, D., Miller, T., Donald, K., & Teodorczuk, A. (2026). AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models. JMIR Nursing, 9, e106133-e106133. https://doi.org/10.2196/106133

Vancouver

Conway A, Hada A, Schluter J, Xu HG, Lowden D, Miller T, et al. AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models. JMIR Nursing. 2026;9:e106133-e106133. doi:10.2196/106133

BibTeX
@article{Conway2026,
  author = {Conway, Aaron and Hada, Adriana and Schluter, Jessica and Xu, Hui Grace and Lowden, Dan and Miller, Tim and Donald, Ken and Teodorczuk, Andrew},
  title = {AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models},
  journal = {JMIR Nursing},
  year = {2026},
  volume = {9},
  pages = {e106133--e106133},
  doi = {10.2196/106133}
}
Start for free →