Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records
The Thoracic and Cardiovascular Surgeon · 01.01.2026 · 74 (S 01 S 01) · S1-S70
DOI: 10.1055/s-0046-1817356 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Thoracic and Cardiovascular Surgeon, erschienen am 01.01.2026. Verfasst von V. Ntinopoulos und 5 weiteren Autor:innen.
Abstract (Originalsprache)
All articles of this category (opens in new window)
Background
Automated extraction of clinical variables from unstructured electronic health records (EHRs) using large language models (LLMs) faces implementation barriers due to data privacy concerns with cloud-based solutions. This study evaluated the diagnostic accuracy of an on-premises open-source LLM for extracting standardized clinical variables from German-language EHR text.
Methods
We performed a retrospective analysis of the de-identified and anonymized medical text of unstructured electronic health records of 50 patients undergoing major cardiac surgery in our department during the year 2023. All texts were in the German language. A 20B parameter open-source LLM was installed on-premises and utilized to process each unstructured medical text. Data extraction comprised information extraction, binary classification, and multiclass classification, involving all variables of the European System for Cardiac Operative Risk Evaluation 2 risk model (EuroSCORE 2). Primary outcomes were overall accuracy, task-specific accuracy, sensitivity, and specificity.
Results
The overall accuracy of the open-source LLM over all 1,100 requested values was 95.1% (1,044 correct values overall). Information extraction accuracy was 100%, binary classification accuracy was 96%, and multiclass classification accuracy was 93.2%. Binary classification sensitivity and specificity were 91.8 and 96.4%, respectively. Multiclass classification sensitivity and specificity were 87.4 and 95%, respectively. Consistency analysis over three same-prompt iterations has shown a Krippendorff’s alpha value of 97.9% (1,076 consistent responses overall).
Conclusion
In this real-world analysis, the 20B parameter open-source LLM demonstrated high diagnostic accuracy for automated extraction of the EuroSCORE 2 variables from unstructured EHR. These findings support the feasibility of on-premises LLM deployment for privacy-compliant clinical data processing, though validation in larger, multicenter cohorts is warranted. Publication History Article published online: 17 February 2026 © 2026. Thieme. All rights reserved. Georg Thieme Verlag KG Oswald-Hesse-Straße 50, 70469 Stuttgart, Germany
Zitieren
APA
Ntinopoulos, V., Biefer, H. R. C., Rosalia, R., Koechlin, L., Rings, L., & Dzemali, O. (2026). Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records. The Thoracic and Cardiovascular Surgeon, 74(S 01 S 01), S1-S70. https://doi.org/10.1055/s-0046-1817356
Vancouver
Ntinopoulos V, Biefer HRC, Rosalia R, Koechlin L, Rings L, Dzemali O. Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records. The Thoracic and Cardiovascular Surgeon. 2026;74(S 01 S 01):S1-S70. doi:10.1055/s-0046-1817356
BibTeX
@article{Ntinopoulos2026,
author = {Ntinopoulos, V. and Biefer, H. Rodriguez Cetina and Rosalia, R. and Koechlin, L. and Rings, L. and Dzemali, O.},
title = {Diagnostic Accuracy of an Open-Source Large Language Model for Automated Data Extraction from Unstructured Electronic Health Records},
journal = {The Thoracic and Cardiovascular Surgeon},
year = {2026},
volume = {74},
number = {S 01 S 01},
pages = {S1--S70},
doi = {10.1055/s-0046-1817356}
}
EUVIMED