EUVIMEDEUVIMED entdecken
Open Access

Labeling Clinical Notes With Large Language Models: Retrospective Benchmarking Study

Pritam Mukherjee, Michael Nazmifar, Vivek Batheja, Ricardo Bigolin Lanfredi, Abhinav Suri, Anisa Prasad, Nicole Tran, Sydney Lewis, Nusrat Rabbee, Ronald M Summers

Journal of Medical Internet Research · 10.11.2026 · 28 · e81708-e81708

DOI: 10.2196/81708 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 10.11.2026. Verfasst von Pritam Mukherjee und 9 weiteren Autor:innen. Beteiligte Einrichtung: Imaging Biomarkers and Computer Aided Diagnosis Lab, Radiology and Imaging Sciences, National Institutes of Health Clinical Center, 10 Center Dr, Bui…. Hauptthemen laut MEDLINE: Benchmarking, Electronic Health Records. Die Arbeit nennt 37 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical notes are rich sources of clinical information and are routinely used in clinical practice to infer past diagnoses and other comorbidities. Large language models (LLMs) may simplify the task by extracting this information from clinical notes.

Objective

The study aims to evaluate the performance of state-of-the-art LLMs on the task of labeling English clinical notes.

Methods

This institutional review board–approved retrospective study comprised 380 clinical notes obtained from 2 institutions in the United States. The notes were manually labeled for the presence or absence of 29 conditions (simplified Elixhauser Comorbidity Index) by 2 human readers—a radiology resident and a medical student under the supervision of a senior radiologist. Forty-six LLMs, including 4 closed-source commercial models from the GPT and Claude families, and 42 open-weights LLMs from the Llama (13 models) and the Qwen (29 models) families were used to label each note. The GPT-o3-mini-high reasoning model was used to parse the free-text model responses to binary yes/no answers. Using the manual labels as the reference standard, the recall, precision, and F 1 -scores for the models were evaluated. The effect of factors such as the length of the notes and the family, size, and release recency of the LLMs was analyzed using generalized linear models.

Results

The top-performing LLMs included the GPT and Claude models, and the larger open-weights models included Llama 3 70B, Llama 3.1 70B, Llama 3.1 405B, Llama 3.3 70B, Qwen 1.5 72B, Qwen 1.5 110B, Qwen 2 72B, and Qwen 2 VL 72B, which achieved F 1 -scores above 70%. The most recent and higher-capacity releases of the LLMs have been improving mainly in their precision. The evaluation of 29 conditions showed a large variance in performance depending on the condition.

Conclusions

In this evaluation with 46 LLMs, larger open-weights LLMs such as the Llama 3.3 70B performed on par with or better than the GPT and Claude models, achieving F 1 -scores above 70%. The performance of open-weights models improved significantly with model size and recency. These results suggest that locally run open-weights LLMs can be potentially used to label clinical notes.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingElectronic Health RecordsLarge Language ModelsRetrospective StudiesHumans

Stichwörter der Autor:innen

benchmarkingclinical noteshealth recordsinformation extractionlarge language models

Autor:innen und Institutionen

  • Imaging Biomarkers and Computer Aided Diagnosis Lab, Radiology and Imaging Sciences, National Institutes of Health Clinical Center, 10 Center Dr, Building 10, Rm 1C224, Bethesda, MD, 20892, United States, 1 301-402-5486.
  • Biostatistics and Clinical Epidemiology Service, National Institutes of Health Clinical Center, Bethesda, MD, United States.

Zitieren

APA

Mukherjee, P., Nazmifar, M., Batheja, V., Bigolin Lanfredi, R., Suri, A., Prasad, A., Tran, N., Lewis, S., Rabbee, N., & Summers, R. M. (2026). Labeling Clinical Notes With Large Language Models: Retrospective Benchmarking Study. Journal of Medical Internet Research, 28, e81708-e81708. https://doi.org/10.2196/81708

Vancouver

Mukherjee P, Nazmifar M, Batheja V, Bigolin Lanfredi R, Suri A, Prasad A, et al. Labeling Clinical Notes With Large Language Models: Retrospective Benchmarking Study. Journal of Medical Internet Research. 2026;28:e81708-e81708. doi:10.2196/81708

BibTeX
@article{Mukherjee2026,
  author = {Mukherjee, Pritam and Nazmifar, Michael and Batheja, Vivek and Bigolin Lanfredi, Ricardo and Suri, Abhinav and Prasad, Anisa and Tran, Nicole and Lewis, Sydney and Rabbee, Nusrat and Summers, Ronald M},
  title = {Labeling Clinical Notes With Large Language Models: Retrospective Benchmarking Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e81708--e81708},
  doi = {10.2196/81708}
}
Kostenlos starten →