A comparison of the suitability of items generated by 4 large language models for pathology continuing certification
Am J Clin Pathol · 2026
DOI: 10.1093/ajcp/aqaf153 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Am J Clin Pathol, 2026. Verfasst von Gary W Procop und 6 weiteren Autor:innen. Beteiligte Einrichtung: The American Board of Pathology, Tampa, FL, United States. Hauptthemen laut MEDLINE: Large Language Models, Certification, Pathology, Education, Medical, Continuing. Die Arbeit nennt 25 Literaturangaben.
Abstract (Originalsprache)
Objective
To evaluate the ability of 4 artificial intelligence large language models (LLMs) to create items that align with the item writing standards of the American Board of Pathology (ABPath) for continuing certification.
Methods
An informatics item writing application was developed and used with prompts based on the ABPath item writing standards. Uniform prompts were used for the LLMs evaluated, with the content of the items generated tailored to the expertise of the reviewing subject matter experts (SMEs). The SMEs were blinded to the identity of the LLM that generated each item. The 14 SMEs graded 4 written items and 4 practical items, with 1 item from each set of 4 generated from each of the LLMs. The 19 questions used for grading concentrated on item anatomy (ie, item structure), accuracy, relevance, and level of item difficulty.
Results
The overall scores for the 4 LLMs for the written items were as follows: Claude, 229 of 266 (86.1%); ChatGPT, 212 of 266 (79.7%); Llama, 175 of 266 (65.8%); and Titan, 162 of 266 (60.9%). The overall scores for the 4 LLMs for the practical items were as follows: Claude, 247 of 266 (92.9%); ChatGPT, 216 of 266 (81.2%); Llama, 175 of 266 (65.8%); and Titan, 151 of 266 (56.8%). Statistically significant differences existed between the LLMs.
Conclusions
We observed significant differences in the ability of the 4 LLMs evaluated to draft items consistent with the ABPath guidelines based on SME scoring. It is important to assess the various LLMs available to determine which model best meets the needs of the user for the proposed task and not to assume equivalence.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The American Board of Pathology, Tampa, FL, United States.
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, IL, United States.
- Department of Pathology, Affiliated Pathologists Medical Group, Rancho Dominguez, CA, United States.
Zitieren
APA
Procop, G. W., Schlinsog, A., Earnest, C., Nayar, R., McCarthy, T., Woodworth, B., & Glassy, E. F. (2026). A comparison of the suitability of items generated by 4 large language models for pathology continuing certification. Am J Clin Pathol. https://doi.org/10.1093/ajcp/aqaf153
Vancouver
Procop GW, Schlinsog A, Earnest C, Nayar R, McCarthy T, Woodworth B, et al. A comparison of the suitability of items generated by 4 large language models for pathology continuing certification. Am J Clin Pathol. 2026. doi:10.1093/ajcp/aqaf153
BibTeX
@article{Procop2026,
author = {Procop, Gary W and Schlinsog, Anthony and Earnest, Colette and Nayar, Ritu and McCarthy, Ty and Woodworth, Bonnie and Glassy, Eric F},
title = {A comparison of the suitability of items generated by 4 large language models for pathology continuing certification},
journal = {Am J Clin Pathol},
year = {2026},
doi = {10.1093/ajcp/aqaf153}
}
EUVIMED