EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Enabling Just-in-Time Clinical Oncology Analysis With Large Language Models: Feasibility and Validation Study Using Unstructured Synthetic Data

Peter May iD, Julian Greß iD, Christoph Seidel iD, Sebastian Sommer iD, Markus K Schuler iD, Sina Nokodian iD, Florian Schröder iD, Johannes Jung iD

JMIR Medical Informatics · 2025

DOI: 10.2196/78332 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Peter May und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine III, School of Medicine and Health, TUM University Hospital, Technical University of Munich, Ismaninger Str. 22, Muni…. Hauptthemen laut MEDLINE: Natural Language Processing, Medical Oncology. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Traditional cancer registries, limited by labor-intensive manual data abstraction and rigid, predefined schemas, often hinder timely and comprehensive oncology research. While large language models (LLMs) have shown promise in automating data extraction, their potential to perform direct, just-in-time (JIT) analysis on unstructured clinical narratives—potentially bypassing intermediate structured databases for many analytical tasks—remains largely unexplored.

Objective

This study aimed to evaluate whether a state-of-the-art LLM (Gemini 2.5 Pro) can enable a JIT clinical oncology analysis paradigm by assessing its ability to (1) perform high-fidelity multiparameter data extraction, (2) answer complex clinical queries directly from raw text, (3) automate multistep survival analyses including executable code generation, and (4) generate novel, clinically plausible hypotheses from free-text documentation.

Methods

A synthetic dataset of 240 unstructured clinical letters from patients with stage IV non–small cell lung cancer (NSCLC), embedding 14 predefined variables, was used. Gemini 2.5 Pro was evaluated on four core JIT capabilities. Performance was measured by using the following metrics: extraction accuracy (compared to human extraction of n=40 letters and across the full n=240 dataset); numerical deviation for direct question answering (n=40 to 240 letters, 5 questions); log-rank P value and Harrell concordance index for LLM-generated versus ground-truth Kaplan-Meier survival analyses (n=160 letters, overall survival and progression-free survival); and correct justification, novelty, and a qualitative evaluation of LLM-generated hypotheses (n=80 and n=160 letters).

Results

For multiparameter extraction from 40 letters, the LLM achieved >99% average accuracy, comparable to human extraction, but in significantly less time (LLM: 3.7 min vs human: 133.8 min). Across the full 240-letter dataset, LLM multiparameter extraction maintained >98% accuracy for most variables. The LLM answered multiconditional clinical queries directly from raw text with a relative deviation rarely exceeding 1.5%, even with up to 240 letters. Crucially, it autonomously performed end-to-end survival analysis, generating text-to-R-code that produced Kaplan-Meier curves statistically indistinguishable from ground truth. Consistent performance was demonstrated on a small validation cohort of 80 synthetic acute myeloid leukemia reports. Stress testing on data with simulated imperfections revealed a key role of a human-in-the-loop to resolve AI-flagged ambiguities. Furthermore, the LLM generated several correctly justified, biologically plausible, and potentially novel hypotheses from datasets up to 80 letters.

Conclusions

This feasibility study demonstrated that a frontier LLM (Gemini 2.5 Pro) can successfully perform high-fidelity data extraction, multiconditional querying, and automated survival analysis directly from unstructured text. These results provide a foundational proof of concept for the JIT clinical analysis approach. However, these findings are confined to synthetic patients, and rigorous validation on real-world clinical data is an essential next step before clinical implementation can be considered.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingMedical OncologyHumansFeasibility StudiesLung NeoplasmsCarcinoma, Non-Small-Cell LungLarge Language Models

Stichwörter der Autor:innen

artificial intelligencecancer registriesclinical oncologyjust-in-time analysislarge language modelsnatural language processingreal-world evidencesurvival analysisunstructured data

Autor:innen und Institutionen

  • Department of Internal Medicine III, School of Medicine and Health, TUM University Hospital, Technical University of Munich, Ismaninger Str. 22, Munich, Germany, 49 89-4140-8753.
  • MPiriQ Science Technologies GmbH, Munich, Germany.
  • Department of Oncology, Hematology and Bone Marrow Transplantation with Division of Pneumology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • MVZ Elisenhof, Munich, Germany.
  • Onkologischer Schwerpunkt am Oskar-Helene Heim, Berlin, Germany.
  • Department of Hematology and Medical Oncology, University Medical Center Göttingen, Göttingen, Germany.

ORCID-Profile: Peter May, Julian Greß, Christoph Seidel, Sebastian Sommer, Markus K Schuler, Sina Nokodian, Florian Schröder, Johannes Jung

Zitieren

APA

May, P., Greß, J., Seidel, C., Sommer, S., Schuler, M. K., Nokodian, S., Schröder, F., & Jung, J. (2025). Enabling Just-in-Time Clinical Oncology Analysis With Large Language Models: Feasibility and Validation Study Using Unstructured Synthetic Data. JMIR Medical Informatics. https://doi.org/10.2196/78332

Vancouver

May P, Greß J, Seidel C, Sommer S, Schuler MK, Nokodian S, et al. Enabling Just-in-Time Clinical Oncology Analysis With Large Language Models: Feasibility and Validation Study Using Unstructured Synthetic Data. JMIR Medical Informatics. 2025. doi:10.2196/78332

BibTeX
@article{May2025,
  author = {May, Peter and Greß, Julian and Seidel, Christoph and Sommer, Sebastian and Schuler, Markus K and Nokodian, Sina and Schröder, Florian and Jung, Johannes},
  title = {Enabling Just-in-Time Clinical Oncology Analysis With Large Language Models: Feasibility and Validation Study Using Unstructured Synthetic Data},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/78332}
}
Kostenlos starten →