EUVIMEDEUVIMED entdecken
Open Access

Using Large Language Model Artificial Intelligence to Enhance Clinical Competency Committee Insight

Mark A Pittman, Carl Ehrett, Mirinda A Gormley, Megan Cifuni, Ray Chahoud, Camiron L Pfennig

AEM Education and Training · 2026-02 · 10 (1)

DOI: 10.1002/aet2.70142 ↗

Kurz gefasst

Fachartikel in der Zeitschrift AEM Education and Training, online erschienen am 20.02.2026. Verfasst von Mark A Pittman und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine Prisma Health Upstate Greenville South Carolina USA. Die Arbeit nennt 11 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence, particularly large language models, has the potential to enhance the evaluation of residents, yet few studies have compared artificial intelligence to humans in the analysis of large volumes of clinical evaluations. The objective of this study was to evaluate the use of large language model (LLM) artificial intelligence (AI) on the evaluation of resident feedback.

Methods

The LLM Llama‐3.1 70B was used on a secure cluster to conduct an analysis of emergency medicine (EM) educator feedback for 31 EM residents at a residency program in the Southeastern United States. The large language model generated assessments summarizing resident strengths, weaknesses, and milestone‐based performance. Members of a clinical competency committee (CCC) blinded to the study hypothesis were surveyed to assess their perceptions on the quality, accuracy, specificity, and usefulness of the AI‐generated content compared to the human‐generated content.

Results

Human‐generated assessments averaged 79 words, while AI‐generated assessments averaged 391 words. Seventy percent of the CCC completed the survey, rating the AI‐generated content more favorably for quality, accuracy, and specificity compared to the human‐generated content. Usefulness of the human‐generated content was reported as good or very good by 71.4%, while usefulness of the AI‐generated content was rated as acceptable (54.1%) or unfavorable (28.6%).

Conclusions

Artificial intelligence can generate EM resident assessments with comparable or superior ratings of quality, accuracy, and specificity relative to human‐generated assessments. This highlights the potential of AI‐driven evaluations to streamline educator review processes, reducing workload without sacrificing integrity.

Autor:innen und Institutionen

  • Department of Emergency Medicine Prisma Health Upstate Greenville South Carolina USA.
  • School of Medicine Greenville University of South Carolina Greenville South Carolina USA.
  • School of Health Research Clemson University Clemson South Carolina USA.
  • Research Computing & Data Clemson University Clemson South Carolina USA.

Zitieren

APA

Pittman, M. A., Ehrett, C., Gormley, M. A., Cifuni, M., Chahoud, R., & Pfennig, C. L. (2026). Using Large Language Model Artificial Intelligence to Enhance Clinical Competency Committee Insight. AEM Education and Training, 10(1). https://doi.org/10.1002/aet2.70142

Vancouver

Pittman MA, Ehrett C, Gormley MA, Cifuni M, Chahoud R, Pfennig CL. Using Large Language Model Artificial Intelligence to Enhance Clinical Competency Committee Insight. AEM Education and Training. 2026;10(1). doi:10.1002/aet2.70142

BibTeX
@article{Pittman2026,
  author = {Pittman, Mark A and Ehrett, Carl and Gormley, Mirinda A and Cifuni, Megan and Chahoud, Ray and Pfennig, Camiron L},
  title = {Using Large Language Model Artificial Intelligence to Enhance Clinical Competency Committee Insight},
  journal = {AEM Education and Training},
  year = {2026},
  volume = {10},
  number = {1},
  doi = {10.1002/aet2.70142}
}
Kostenlos starten →