Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy
The Journal of Clinical Pharmacology · 2024-09 · 64 (9) · 1095-1100
DOI: 10.1002/jcph.2443 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Journal of Clinical Pharmacology, online erschienen am 16.04.2024. Verfasst von Merel van Nuland und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Clinical Pharmacy, Tergooi Medical Center, Hilversum, The Netherlands. Hauptthemen laut MEDLINE: Pharmacists. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 11 Literaturangaben.
Abstract (Originalsprache)
ChatGPT is a language model that was trained on a large dataset including medical literature. Several studies have described the performance of ChatGPT on medical exams. In this study, we examine its performance in answering factual knowledge questions regarding clinical pharmacy. Questions were obtained from a Dutch application that features multiple‐choice questions to maintain a basic knowledge level for clinical pharmacists. In total, 264 clinical pharmacy‐related questions were presented to ChatGPT and responses were evaluated for accuracy, concordance, quality of the substantiation, and reproducibility. Accuracy was defined as the correctness of the answer, and results were compared to the overall score by pharmacists over 2022. Responses were marked concordant if no contradictions were present. The quality of the substantiation was graded by two independent pharmacists using a 4‐point scale. Reproducibility was established by presenting questions multiple times and on various days. ChatGPT yielded accurate responses for 79% of the questions, surpassing pharmacists' accuracy of 66%. Concordance was 95%, and the quality of the substantiation was deemed good or excellent for 73% of the questions. Reproducibility was consistently high, both within day and between days (>92%), as well as across different users. ChatGPT demonstrated a higher accuracy and reproducibility to factual knowledge questions related to clinical pharmacy practice than pharmacists. Consequently, we posit that ChatGPT could serve as a valuable resource to pharmacists. We hope the technology will further improve, which may lead to enhanced future performance.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Clinical Pharmacy, Tergooi Medical Center, Hilversum, The Netherlands.
- Department of Clinical Pharmacy, Amphia Hospital, Breda, The Netherlands.
- Department of Clinical Pharmacy, Leeuwarden Medical Center, Leeuwarden, The Netherlands.
- Department of Clinical Pharmacy, Haga Hospital, The Hague, The Netherlands.
- Department of Clinical Pharmacy, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
- Division of Pharmacoepidemiology and Clinical Pharmacology, Department of Pharmaceutical Sciences, Faculty of Science, Utrecht Institute for Pharmaceutical Sciences (UIPS), Utrecht University, Utrecht, The Netherlands.
ORCID-Profile: Merel van Nuland
Zitieren
APA
van Nuland, M., Erdogan, A., Aςar, C., Contrucci, R., Hilbrants, S., Maanach, L., Egberts, T., & van der Linden, P. D. (2024). Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy. The Journal of Clinical Pharmacology, 64(9), 1095-1100. https://doi.org/10.1002/jcph.2443
Vancouver
van Nuland M, Erdogan A, Aςar C, Contrucci R, Hilbrants S, Maanach L, et al. Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy. The Journal of Clinical Pharmacology. 2024;64(9):1095-1100. doi:10.1002/jcph.2443
BibTeX
@article{vanNuland2024,
author = {van Nuland, Merel and Erdogan, Abdullah and Aςar, Cenkay and Contrucci, Ramon and Hilbrants, Sven and Maanach, Lamyae and Egberts, Toine and van der Linden, Paul D},
title = {Performance of ChatGPT on Factual Knowledge Questions Regarding Clinical Pharmacy},
journal = {The Journal of Clinical Pharmacology},
year = {2024},
volume = {64},
number = {9},
pages = {1095--1100},
doi = {10.1002/jcph.2443}
}
EUVIMED