Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study
JMIR Formative Research · 2025
DOI: 10.2196/64544 ↗At a glance
Kohortenstudie in JMIR Formative Research, 2025. Written by Denise Lee and 6 co-authors. Institution: Department of Surgery, Icahn School of Medicine at Mount Sinai, 10 Union Square East, Suite 2L, New York, NY, 10003, United States, 1 212 241 2891. Major MEDLINE topics: Natural Language Processing, Pathology, Surgical, Information Storage and Retrieval, Data Mining. Cited 35 times so far (Crossref); 21 references. The full text is freely available in PubMed Central. Funded by: NCATS NIH HHS.
Abstract (original language)
Background
Popularized by ChatGPT, large language models (LLMs) are poised to transform the scalability of clinical natural language processing (NLP) downstream tasks such as medical question answering (MQA) and automated data extraction from clinical narrative reports. However, the use of LLMs in the health care setting is limited by cost, computing power, and patient privacy concerns. Specifically, as interest in LLM-based clinical applications grows, regulatory safeguards must be established to avoid exposure of patient data through the public domain. The use of open-source LLMs deployed behind institutional firewalls may ensure the protection of private patient data. In this study, we evaluated the extraction performance of a locally deployed LLM for automated MQA from surgical pathology reports.
Objective
We compared the performance of human reviewers and a locally deployed LLM tasked with extracting key histologic and staging information from surgical pathology reports.
Methods
A total of 84 thyroid cancer surgical pathology reports were assessed by two independent reviewers and the open-source FastChat-T5 3B-parameter LLM using institutional computing resources. Longer text reports were split into 1200-character-long segments, followed by conversion to embeddings. Three segments with the highest similarity scores were integrated to create the final context for the LLM. The context was then made part of the question it was directed to answer. Twelve medical questions for staging and thyroid cancer recurrence risk data extraction were formulated and answered for each report. The time to respond and concordance of answers were evaluated. The concordance rate for each pairwise comparison (human-LLM and human-human) was calculated as the total number of concordant answers divided by the total number of answers for each of the 12 questions. The average concordance rate and associated error of all questions were tabulated for each pairwise comparison and evaluated with two-sided t tests.
Results
Out of a total of 1008 questions answered, reviewers 1 and 2 had an average (SD) concordance rate of responses of 99% (1%; 999/1008 responses). The LLM was concordant with reviewers 1 and 2 at an overall average (SD) rate of 89% (7%; 896/1008 responses) and 89% (7.2%; 903/1008 responses). The overall time to review and answer questions for all reports was 170.7, 115, and 19.56 minutes for Reviewers 1, 2, and the LLM, respectively.
Conclusions
The locally deployed LLM can be used for MQA with considerable time-saving and acceptable accuracy in responses. Prompt engineering and fine-tuning may further augment automated data extraction from clinical narratives for the provision of real-time, essential clinical insights.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Surgery, Icahn School of Medicine at Mount Sinai, 10 Union Square East, Suite 2L, New York, NY, 10003, United States, 1 212 241 2891.
- Charles Bronfman Institute for Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
- Department of Statistics and Data Science, Cornell University, Ithaca, NY, United States.
ORCID profiles: Denise Lee, Akhil Vaid, Kartikeya M Menon, Robert Freeman, David S Matteson, Michael L Marin, Girish N Nadkarni
Funding
- NCATS NIH HHS
Cite
APA
Lee, D., Vaid, A., Menon, K. M., Freeman, R., Matteson, D. S., Marin, M. L., & Nadkarni, G. N. (2025). Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study. JMIR Formative Research. https://doi.org/10.2196/64544
Vancouver
Lee D, Vaid A, Menon KM, Freeman R, Matteson DS, Marin ML, et al. Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study. JMIR Formative Research. 2025. doi:10.2196/64544
BibTeX
@article{Lee2025,
author = {Lee, Denise and Vaid, Akhil and Menon, Kartikeya M and Freeman, Robert and Matteson, David S and Marin, Michael L and Nadkarni, Girish N},
title = {Using Large Language Models to Automate Data Extraction From Surgical Pathology Reports: Retrospective Cohort Study},
journal = {JMIR Formative Research},
year = {2025},
doi = {10.2196/64544}
}
EUVIMED