A Comparative Study of Artificial Intelligence and Faculty Rubric‐Based Grading of Pharmacy Student Writing Assignments in an Evidence‐Based Medicine Course
JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · 2026-07 · 9 (7)
DOI: 10.1002/jac5.70249 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, online erschienen am 29.06.2026. Verfasst von Jennifer Phillips und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of Illinois Chicago Retzky College of Pharmacy, Chicago, Illinois, USA. Hauptthemen laut MEDLINE: Writing, Students, Pharmacy, Evidence-Based Medicine, Educational Measurement. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 4 Literaturangaben. Gefördert von: American College of Clinical Pharmacy.
Abstract (Originalsprache)
Background
Generative artificial intelligence (AI) tools are increasingly used in health professional education, including for automated writing evaluation (AWE). While AI‐based grading may reduce faculty workload and variability, its alignment with human grading remains unclear. This study compared rubric‐based grading of pharmacy student drug information papers by faculty and ChatGPT (Open AI, San Francisco, CA).
Methods
We conducted a retrospective comparative analysis of 159 de‐identified assignments from a required evidence‐based medicine course. Faculty‐assigned grades were paired with scores generated by a custom generative pretrained transformer (GPT) configured with the assignment rubric. The primary outcome was the difference in mean total scores; secondary outcomes included rubric section‐level scores, variability, and agreement metrics. Paired t ‐tests, Lin's concordance correlation coefficient, and Cohen's weighted kappa were used for analysis.
Results
Aggregate mean total scores were higher for ChatGPT versus faculty (51.3 vs. 49; p = 0.0004, mean difference, 2.3; 95% confidence interval, 1.015–3.595), with differences in five of seven rubric sections. Faculty scores exhibited greater variability than ChatGPT scores (standard deviation, 7.4 vs. 3.7). Concordance between faculty and AI grading at the individual student level was poor (Lin's coefficient for total score = 0.06; kappa for overall grade = 0.03). ChatGPT also assigned more “B” grades and fewer failing grades compared with faculty.
Conclusion
AI‐based grading produced similar aggregate letter grades and reduced variability but demonstrated poor agreement with faculty scores at the individual level. AI grading may complement, not replace, faculty evaluation for assignments requiring critical appraisal, but further studies are needed.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University of Illinois Chicago Retzky College of Pharmacy, Chicago, Illinois, USA.
ORCID-Profile: Jennifer Phillips, Katherine Sarna, Faria Munir, Heather Ipema
Förderung
- American College of Clinical Pharmacy
Zitieren
APA
Phillips, J., Sarna, K., Munir, F., & Ipema, H. (2026). A Comparative Study of Artificial Intelligence and Faculty Rubric‐Based Grading of Pharmacy Student Writing Assignments in an Evidence‐Based Medicine Course. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, 9(7). https://doi.org/10.1002/jac5.70249
Vancouver
Phillips J, Sarna K, Munir F, Ipema H. A Comparative Study of Artificial Intelligence and Faculty Rubric‐Based Grading of Pharmacy Student Writing Assignments in an Evidence‐Based Medicine Course. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. 2026;9(7). doi:10.1002/jac5.70249
BibTeX
@article{Phillips2026,
author = {Phillips, Jennifer and Sarna, Katherine and Munir, Faria and Ipema, Heather},
title = {A Comparative Study of Artificial Intelligence and Faculty Rubric‐Based Grading of Pharmacy Student Writing Assignments in an Evidence‐Based Medicine Course},
journal = {JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY},
year = {2026},
volume = {9},
number = {7},
doi = {10.1002/jac5.70249}
}
EUVIMED