EUVIMEDEUVIMED entdecken
Vergleichsstudie

A comparison of the suitability of items generated by 4 large language models for pathology continuing certification

Gary W Procop, Anthony Schlinsog, Colette Earnest, Ritu Nayar, Ty McCarthy, Bonnie Woodworth, Eric F Glassy

Am J Clin Pathol · 2026

DOI: 10.1093/ajcp/aqaf153 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Am J Clin Pathol, 2026. Verfasst von Gary W Procop und 6 weiteren Autor:innen. Beteiligte Einrichtung: The American Board of Pathology, Tampa, FL, United States. Hauptthemen laut MEDLINE: Large Language Models, Certification, Pathology, Education, Medical, Continuing. Die Arbeit nennt 25 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate the ability of 4 artificial intelligence large language models (LLMs) to create items that align with the item writing standards of the American Board of Pathology (ABPath) for continuing certification.

Methods

An informatics item writing application was developed and used with prompts based on the ABPath item writing standards. Uniform prompts were used for the LLMs evaluated, with the content of the items generated tailored to the expertise of the reviewing subject matter experts (SMEs). The SMEs were blinded to the identity of the LLM that generated each item. The 14 SMEs graded 4 written items and 4 practical items, with 1 item from each set of 4 generated from each of the LLMs. The 19 questions used for grading concentrated on item anatomy (ie, item structure), accuracy, relevance, and level of item difficulty.

Results

The overall scores for the 4 LLMs for the written items were as follows: Claude, 229 of 266 (86.1%); ChatGPT, 212 of 266 (79.7%); Llama, 175 of 266 (65.8%); and Titan, 162 of 266 (60.9%). The overall scores for the 4 LLMs for the practical items were as follows: Claude, 247 of 266 (92.9%); ChatGPT, 216 of 266 (81.2%); Llama, 175 of 266 (65.8%); and Titan, 151 of 266 (56.8%). Statistically significant differences existed between the LLMs.

Conclusions

We observed significant differences in the ability of the 4 LLMs evaluated to draft items consistent with the ABPath guidelines based on SME scoring. It is important to assess the various LLMs available to determine which model best meets the needs of the user for the proposed task and not to assume equivalence.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsCertificationPathologyEducation, Medical, ContinuingArtificial IntelligenceHumansWriting

Stichwörter der Autor:innen

artificial intelligenceitem writinglarge language models

Autor:innen und Institutionen

  • The American Board of Pathology, Tampa, FL, United States.
  • Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, IL, United States.
  • Department of Pathology, Affiliated Pathologists Medical Group, Rancho Dominguez, CA, United States.

Zitieren

APA

Procop, G. W., Schlinsog, A., Earnest, C., Nayar, R., McCarthy, T., Woodworth, B., & Glassy, E. F. (2026). A comparison of the suitability of items generated by 4 large language models for pathology continuing certification. Am J Clin Pathol. https://doi.org/10.1093/ajcp/aqaf153

Vancouver

Procop GW, Schlinsog A, Earnest C, Nayar R, McCarthy T, Woodworth B, et al. A comparison of the suitability of items generated by 4 large language models for pathology continuing certification. Am J Clin Pathol. 2026. doi:10.1093/ajcp/aqaf153

BibTeX
@article{Procop2026,
  author = {Procop, Gary W and Schlinsog, Anthony and Earnest, Colette and Nayar, Ritu and McCarthy, Ty and Woodworth, Bonnie and Glassy, Eric F},
  title = {A comparison of the suitability of items generated by 4 large language models for pathology continuing certification},
  journal = {Am J Clin Pathol},
  year = {2026},
  doi = {10.1093/ajcp/aqaf153}
}
Kostenlos starten →