Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education
Journal of Medical Internet Research · 2025-07-15 · 27 · e74299
DOI: 10.2196/74299 ↗At a glance
Validierungsstudie in Journal of Medical Internet Research, published online on 2025-07-15. Written by Chung Man Ho and 8 co-authors. Institution: Neurosurgery Department, Queen Elizabeth Hospital, Kowloon, China (Hong Kong). Major MEDLINE topics: Patient Education as Topic, Artificial Intelligence, Language, Neurosurgery. Cited 20 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.
Topics: Künstliche Intelligenz in der Pflege, Selbstmanagement bei chronischen Erkrankungen
Abstract (original language)
Background
Perioperative education is crucial for optimizing outcomes in neuroendovascular procedures, where inadequate understanding can heighten patient anxiety and hinder care plan adherence. Current education models, reliant on traditional consultations and printed materials, often lack scalability and personalization. Artificial intelligence (AI)–powered chatbots have demonstrated efficacy in various health care contexts; however, their role in neuroendovascular perioperative support remains underexplored. Given the complexity of neuroendovascular procedures and the need for continuous, tailored patient education, AI chatbots have the potential to offer tailored perioperative guidance to improve patient education in this specialty.
Objective
We aimed to develop, validate, and assess NeuroBot, an AI-driven system that uses large language models (LLMs) with retrieval-augmented generation to deliver timely, accurate, and evidence-based responses to patient inquiries in neurosurgery, ultimately improving the effectiveness of patient education.
Methods
A mixed methods approach was used, consisting of 3 phases. In the first phase, internal validation, we compared the performance of Assistants API, ChatGPT, and Qwen by evaluating their responses to 306 bilingual neuroendovascular-related questions. The accuracy, relevance, and completeness of the responses were evaluated using a Likert scale; statistical analyses included ANOVA and paired t tests. In the second phase, external validation, 10 neurosurgical experts rated the responses generated by NeuroBot using the same evaluation metrics applied in the internal validation phase. The consistency of their ratings was measured using the intraclass correlation coefficient. Finally, in the third phase, a qualitative study was conducted through interviews with 18 health care providers, which helped identify key themes related to the NeuroBot’s usability and perceived benefits. Thematic analysis was performed using NVivo and interrater reliability was confirmed through Cohen κ.
Results
The Assistants API outperformed both ChatGPT and Qwen, achieving a mean accuracy score of 5.28 out of 6 (95% CI 5.21-5.35), with a statistically significant result (P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Neurosurgery Department, Queen Elizabeth Hospital, Kowloon, China (Hong Kong).
- Department of Electrical and Electronic Engineering, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
- Faculty of Medicine, Chinese University of Hong Kong, New Territories, China (Hong Kong).
- School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Central and Western District, China (Hong Kong).
- Center for Smart Health, School of Nursing, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
- School of Nursing, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
ORCID profiles: Chung Man Ho, Shaowei Guan, Prudence Kwan-Lam Mok, Candice Hw Lam, Wai Ying Ho, Calvin Hoi-Kwan Mak, Harry Qin, Arkers Kwan Ching Wong, Vivian Hui
Cite
APA
Ho, C. M., Guan, S., Mok, P. K., Lam, C. H., Ho, W. Y., Mak, C. H., Qin, H., Wong, A. K. C., & Hui, V. (2025). Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education. Journal of Medical Internet Research, 27, e74299. https://doi.org/10.2196/74299
Vancouver
Ho CM, Guan S, Mok PK, Lam CH, Ho WY, Mak CH, et al. Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education. Journal of Medical Internet Research. 2025;27:e74299. doi:10.2196/74299
BibTeX
@article{Ho2025,
author = {Ho, Chung Man and Guan, Shaowei and Mok, Prudence Kwan-Lam and Lam, Candice Hw and Ho, Wai Ying and Mak, Calvin Hoi-Kwan and Qin, Harry and Wong, Arkers Kwan Ching and Hui, Vivian},
title = {Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e74299},
doi = {10.2196/74299}
}
EUVIMED