EUVIMEDEUVIMED entdecken
Crossref

Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records

V. Ntinopoulos, H. Rodriguez Cetina Biefer, R. Rosalia, L. Koechlin, L. Rings, O. Dzemali

The Thoracic and Cardiovascular Surgeon · 01.01.2026 · 74 (S 01 S 01) · S1-S70

DOI: 10.1055/s-0046-1817356 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The Thoracic and Cardiovascular Surgeon, erschienen am 01.01.2026. Verfasst von V. Ntinopoulos und 5 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

All articles of this category (opens in new window)

Background

Automated extraction of clinical variables from unstructured electronic health records (EHRs) using large language models (LLMs) faces implementation barriers due to data privacy concerns with cloud-based solutions. This study evaluated the diagnostic accuracy of an on-premises open-source LLM for extracting standardized clinical variables from German-language EHR text.

Methods

We performed a retrospective analysis of the de-identified and anonymized medical text of unstructured electronic health records of 50 patients undergoing major cardiac surgery in our department during the year 2023. All texts were in the German language. A 20B parameter open-source LLM was installed on-premises and utilized to process each unstructured medical text. Data extraction comprised information extraction, binary classification, and multiclass classification, involving all variables of the European System for Cardiac Operative Risk Evaluation 2 risk model (EuroSCORE 2). Primary outcomes were overall accuracy, task-specific accuracy, sensitivity, and specificity.

Results

The overall accuracy of the open-source LLM over all 1,100 requested values was 95.1% (1,044 correct values overall). Information extraction accuracy was 100%, binary classification accuracy was 96%, and multiclass classification accuracy was 93.2%. Binary classification sensitivity and specificity were 91.8 and 96.4%, respectively. Multiclass classification sensitivity and specificity were 87.4 and 95%, respectively. Consistency analysis over three same-prompt iterations has shown a Krippendorff’s alpha value of 97.9% (1,076 consistent responses overall).

Conclusion

In this real-world analysis, the 20B parameter open-source LLM demonstrated high diagnostic accuracy for automated extraction of the EuroSCORE 2 variables from unstructured EHR. These findings support the feasibility of on-premises LLM deployment for privacy-compliant clinical data processing, though validation in larger, multicenter cohorts is warranted. Publication History Article published online: 17 February 2026 © 2026. Thieme. All rights reserved. Georg Thieme Verlag KG Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany

Zitieren

APA

Ntinopoulos, V., Biefer, H. R. C., Rosalia, R., Koechlin, L., Rings, L., & Dzemali, O. (2026). Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records. The Thoracic and Cardiovascular Surgeon, 74(S 01 S 01), S1-S70. https://doi.org/10.1055/s-0046-1817356

Vancouver

Ntinopoulos V, Biefer HRC, Rosalia R, Koechlin L, Rings L, Dzemali O. Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records. The Thoracic and Cardiovascular Surgeon. 2026;74(S 01 S 01):S1-S70. doi:10.1055/s-0046-1817356

BibTeX
@article{Ntinopoulos2026,
  author = {Ntinopoulos, V. and Biefer, H. Rodriguez Cetina and Rosalia, R. and Koechlin, L. and Rings, L. and Dzemali, O.},
  title = {Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records},
  journal = {The Thoracic and Cardiovascular Surgeon},
  year = {2026},
  volume = {74},
  number = {S 01 S 01},
  pages = {S1--S70},
  doi = {10.1055/s-0046-1817356}
}
Kostenlos starten →