EUVIMEDSkopri EUVIMED
Vergleichsstudie

ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study

Gautham Arun, Vivek Perumal, Francis Paul John Bato Urias, Yan En Ler, Bryan Wen Tao Tan, Ranganath Vallabhajosyula, Emmanuel Tan, Olivia Ng iD, Kian Bee Ng, Sreenivasulu Reddy Mogali iD

Anatomical Sciences Education · 2024-10 · 17 (7) · 1396-1405

DOI: 10.1002/ase.2502 ↗

At a glance

Vergleichsstudie in Anatomical Sciences Education, published online on 2024-08-21. Written by Gautham Arun and 9 co-authors. Institution: Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore. Major MEDLINE topics: Anatomy. Cited 64 times so far (Crossref); 45 references. Funded by: Special Project Grant from the Medical Education, Research and Scholarship Unit, Lee Kong Chian School of Medicine.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT‐generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI‐driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context‐aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy ( n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5‐point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher ( U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.

Subjects

MeSH terms (MEDLINE)

AnatomyPilot ProjectsHumansArtificial IntelligenceComputer-Assisted InstructionEducational MeasurementLearningSoftware

Author keywords

ChatGPTanatomical educationartificial intelligencechatbotpersonalized learningretrieval augmented generation

Authors and institutions

  • Lee Kong Chian School of Medicine, Nanyang Technological University Singapore, Singapore, Singapore.
  • Singapore Polytechnic, Singapore, Singapore.

ORCID profiles: Olivia Ng, Sreenivasulu Reddy Mogali

Funding

  • Special Project Grant from the Medical Education, Research and Scholarship Unit, Lee Kong Chian School of Medicine

Cite

APA

Arun, G., Perumal, V., Urias, F. P. J. B., Ler, Y. E., Tan, B. W. T., Vallabhajosyula, R., Tan, E., Ng, O., Ng, K. B., & Mogali, S. R. (2024). ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study. Anatomical Sciences Education, 17(7), 1396-1405. https://doi.org/10.1002/ase.2502

Vancouver

Arun G, Perumal V, Urias FPJB, Ler YE, Tan BWT, Vallabhajosyula R, et al. ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study. Anatomical Sciences Education. 2024;17(7):1396-1405. doi:10.1002/ase.2502

BibTeX
@article{Arun2024,
  author = {Arun, Gautham and Perumal, Vivek and Urias, Francis Paul John Bato and Ler, Yan En and Tan, Bryan Wen Tao and Vallabhajosyula, Ranganath and Tan, Emmanuel and Ng, Olivia and Ng, Kian Bee and Mogali, Sreenivasulu Reddy},
  title = {ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study},
  journal = {Anatomical Sciences Education},
  year = {2024},
  volume = {17},
  number = {7},
  pages = {1396--1405},
  doi = {10.1002/ase.2502}
}
Start for free →