EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY-NC-ND 4.0

Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements

Brittany Curry, Amrit Kirpalani, Mia Remington, Tamara Van Hooren, Ye Shen, Erin R Peebles

Canadian Medical Education Journal · 2025

DOI: 10.36834/cmej.81345 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Canadian Medical Education Journal, 2025. Verfasst von Brittany Curry und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics, University of British Columbia, British Columbia, Canada. Hauptthemen laut MEDLINE: Pediatrics, Internship and Residency, School Admission Criteria, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Personal statements play a large role in pediatric residency applications, providing insights into candidates’ motivations, experiences, and fit for the program. With large language models (LLMs) such as Chat Generative Pre-trained Transformer (ChatGPT), concerns have arisen regarding how this may influence the authenticity of statements in evaluating candidates. This study investigates the efficacy and perceived authenticity of LLM-generated personal statements compared to human-generated statements in residency applications.

Methods

We conducted a blinded study comparing 30 ChatGPT-generated personal statements with 30 human-written statements. Four pediatric faculty raters assessed each statement using a standardized 10-point rubric. We analyzed the data using linear mixed-effects models, a chi-square sensitivity analysis, an evaluation of rater accuracy in identifying statement origin as well as consistency of scores amongst raters using intraclass correlation coefficients (ICC).

Results

There was no significant difference in mean scores between AI and human-written statements. Raters could only identify the source of a letter (AI or human) with 59% accuracy. There was considerable disagreement in scores between raters as indicated by negative ICCs.

Conclusions

AI-generated statements were rated similarly to human-authored statements and were indistinguishable by reviewers, highlighting the sophistication of these LLM models and the challenge in detecting their use. Furthermore, scores varied substantially between reviewers. As AI becomes increasingly used in application processes, it is imperative to examine its implications in the overall evaluation of applicants.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PediatricsInternship and ResidencySchool Admission CriteriaArtificial IntelligenceHumansGenerative Artificial IntelligenceLarge Language ModelsFaculty, Medical

Autor:innen und Institutionen

  • Department of Pediatrics, University of British Columbia, British Columbia, Canada.
  • Department of Pediatrics, Western University, Ontario, Canada.
  • BC Children's Hospital Research Institute, British Columbia, Canada.
  • Centre for Health Education Scholarship, University of British Columbia, British Columbia, Canada.

Zitieren

APA

Curry, B., Kirpalani, A., Remington, M., Van Hooren, T., Shen, Y., & Peebles, E. R. (2025). Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements. Canadian Medical Education Journal. https://doi.org/10.36834/cmej.81345

Vancouver

Curry B, Kirpalani A, Remington M, Van Hooren T, Shen Y, Peebles ER. Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements. Canadian Medical Education Journal. 2025. doi:10.36834/cmej.81345

BibTeX
@article{Curry2025,
  author = {Curry, Brittany and Kirpalani, Amrit and Remington, Mia and Van Hooren, Tamara and Shen, Ye and Peebles, Erin R},
  title = {Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements},
  journal = {Canadian Medical Education Journal},
  year = {2025},
  doi = {10.36834/cmej.81345}
}
Kostenlos starten →