ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study
Anatomical Sciences Education · 2024-10 · 17 (7) · 1396-1405
DOI: 10.1002/ase.2502 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Anatomical Sciences Education, online erschienen am 21.08.2024. Verfasst von Gautham Arun und 9 weiteren Autor:innen. Beteiligte Einrichtung: Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore. Hauptthemen laut MEDLINE: Anatomy. Die Arbeit wurde bisher 64-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: Special Project Grant from the Medical Education, Research and Scholarship Unit, Lee Kong Chian School of Medicine.
Abstract (Originalsprache)
Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT‐generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI‐driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context‐aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy ( n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5‐point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher ( U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore.
- Singapore Polytechnic, Singapore, Singapore.
ORCID-Profile: Olivia Ng, Sreenivasulu Reddy Mogali
Förderung
- Special Project Grant from the Medical Education, Research and Scholarship Unit, Lee Kong Chian School of Medicine
Zitieren
APA
Arun, G., Perumal, V., Urias, F. P. J. B., Ler, Y. E., Tan, B. W. T., Vallabhajosyula, R., Tan, E., Ng, O., Ng, K. B., & Mogali, S. R. (2024). ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study. Anatomical Sciences Education, 17(7), 1396-1405. https://doi.org/10.1002/ase.2502
Vancouver
Arun G, Perumal V, Urias FPJB, Ler YE, Tan BWT, Vallabhajosyula R, et al. ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study. Anatomical Sciences Education. 2024;17(7):1396-1405. doi:10.1002/ase.2502
BibTeX
@article{Arun2024,
author = {Arun, Gautham and Perumal, Vivek and Urias, Francis Paul John Bato and Ler, Yan En and Tan, Bryan Wen Tao and Vallabhajosyula, Ranganath and Tan, Emmanuel and Ng, Olivia and Ng, Kian Bee and Mogali, Sreenivasulu Reddy},
title = {ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study},
journal = {Anatomical Sciences Education},
year = {2024},
volume = {17},
number = {7},
pages = {1396--1405},
doi = {10.1002/ase.2502}
}
EUVIMED