Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements
Canadian Medical Education Journal · 2025
DOI: 10.36834/cmej.81345 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Canadian Medical Education Journal, 2025. Verfasst von Brittany Curry und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pediatrics, University of British Columbia, British Columbia, Canada. Hauptthemen laut MEDLINE: Pediatrics, Internship and Residency, School Admission Criteria, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Background
Personal statements play a large role in pediatric residency applications, providing insights into candidates’ motivations, experiences, and fit for the program. With large language models (LLMs) such as Chat Generative Pre-trained Transformer (ChatGPT), concerns have arisen regarding how this may influence the authenticity of statements in evaluating candidates. This study investigates the efficacy and perceived authenticity of LLM-generated personal statements compared to human-generated statements in residency applications.
Methods
We conducted a blinded study comparing 30 ChatGPT-generated personal statements with 30 human-written statements. Four pediatric faculty raters assessed each statement using a standardized 10-point rubric. We analyzed the data using linear mixed-effects models, a chi-square sensitivity analysis, an evaluation of rater accuracy in identifying statement origin as well as consistency of scores amongst raters using intraclass correlation coefficients (ICC).
Results
There was no significant difference in mean scores between AI and human-written statements. Raters could only identify the source of a letter (AI or human) with 59% accuracy. There was considerable disagreement in scores between raters as indicated by negative ICCs.
Conclusions
AI-generated statements were rated similarly to human-authored statements and were indistinguishable by reviewers, highlighting the sophistication of these LLM models and the challenge in detecting their use. Furthermore, scores varied substantially between reviewers. As AI becomes increasingly used in application processes, it is imperative to examine its implications in the overall evaluation of applicants.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Pediatrics, University of British Columbia, British Columbia, Canada.
- Department of Pediatrics, Western University, Ontario, Canada.
- BC Children's Hospital Research Institute, British Columbia, Canada.
- Centre for Health Education Scholarship, University of British Columbia, British Columbia, Canada.
Zitieren
APA
Curry, B., Kirpalani, A., Remington, M., Van Hooren, T., Shen, Y., & Peebles, E. R. (2025). Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements. Canadian Medical Education Journal. https://doi.org/10.36834/cmej.81345
Vancouver
Curry B, Kirpalani A, Remington M, Van Hooren T, Shen Y, Peebles ER. Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements. Canadian Medical Education Journal. 2025. doi:10.36834/cmej.81345
BibTeX
@article{Curry2025,
author = {Curry, Brittany and Kirpalani, Amrit and Remington, Mia and Van Hooren, Tamara and Shen, Ye and Peebles, Erin R},
title = {Re-evaluating the role of personal statements in pediatric residency admissions in the era of artificial intelligence: comparing faculty ratings of human and AI-generated statements},
journal = {Canadian Medical Education Journal},
year = {2025},
doi = {10.36834/cmej.81345}
}
EUVIMED