EUVIMEDDescoperă EUVIMED
ValidierungsstudieMixed-Methods-StudieOpen access

Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education

Chung Man Ho iD, Shaowei Guan iD, Prudence Kwan-Lam Mok iD, Candice Hw Lam iD, Wai Ying Ho iD, Calvin Hoi-Kwan Mak iD, Harry Qin iD, Arkers Kwan Ching Wong iD, Vivian Hui iD

Journal of Medical Internet Research · 2025-07-15 · 27 · e74299

DOI: 10.2196/74299 ↗

At a glance

Validierungsstudie in Journal of Medical Internet Research, published online on 2025-07-15. Written by Chung Man Ho and 8 co-authors. Institution: Neurosurgery Department, Queen Elizabeth Hospital, Kowloon, China (Hong Kong). Major MEDLINE topics: Patient Education as Topic, Artificial Intelligence, Language, Neurosurgery. Cited 20 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Background

Perioperative education is crucial for optimizing outcomes in neuroendovascular procedures, where inadequate understanding can heighten patient anxiety and hinder care plan adherence. Current education models, reliant on traditional consultations and printed materials, often lack scalability and personalization. Artificial intelligence (AI)–powered chatbots have demonstrated efficacy in various health care contexts; however, their role in neuroendovascular perioperative support remains underexplored. Given the complexity of neuroendovascular procedures and the need for continuous, tailored patient education, AI chatbots have the potential to offer tailored perioperative guidance to improve patient education in this specialty.

Objective

We aimed to develop, validate, and assess NeuroBot, an AI-driven system that uses large language models (LLMs) with retrieval-augmented generation to deliver timely, accurate, and evidence-based responses to patient inquiries in neurosurgery, ultimately improving the effectiveness of patient education.

Methods

A mixed methods approach was used, consisting of 3 phases. In the first phase, internal validation, we compared the performance of Assistants API, ChatGPT, and Qwen by evaluating their responses to 306 bilingual neuroendovascular-related questions. The accuracy, relevance, and completeness of the responses were evaluated using a Likert scale; statistical analyses included ANOVA and paired t tests. In the second phase, external validation, 10 neurosurgical experts rated the responses generated by NeuroBot using the same evaluation metrics applied in the internal validation phase. The consistency of their ratings was measured using the intraclass correlation coefficient. Finally, in the third phase, a qualitative study was conducted through interviews with 18 health care providers, which helped identify key themes related to the NeuroBot’s usability and perceived benefits. Thematic analysis was performed using NVivo and interrater reliability was confirmed through Cohen κ.

Results

The Assistants API outperformed both ChatGPT and Qwen, achieving a mean accuracy score of 5.28 out of 6 (95% CI 5.21-5.35), with a statistically significant result (P

Subjects

MeSH terms (MEDLINE)

Patient Education as TopicArtificial IntelligenceLanguageNeurosurgeryNeurosurgical ProceduresHumansFemaleMaleMiddle AgedLarge Language ModelsGenerative Artificial Intelligence

Author keywords

artificial intelligencechatbotdigital healthlarge language modelneurosurgerypatient educationpatient-centered careperioperative careretrieval-augmented generation

Authors and institutions

  • Neurosurgery Department, Queen Elizabeth Hospital, Kowloon, China (Hong Kong).
  • Department of Electrical and Electronic Engineering, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
  • Faculty of Medicine, Chinese University of Hong Kong, New Territories, China (Hong Kong).
  • School of Clinical Medicine, Li Ka Shing Faculty of Medicine, University of Hong Kong, Central and Western District, China (Hong Kong).
  • Center for Smart Health, School of Nursing, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).
  • School of Nursing, Hong Kong Polytechnic University, Kowloon, China (Hong Kong).

ORCID profiles: Chung Man Ho, Shaowei Guan, Prudence Kwan-Lam Mok, Candice Hw Lam, Wai Ying Ho, Calvin Hoi-Kwan Mak, Harry Qin, Arkers Kwan Ching Wong, Vivian Hui

Cite

APA

Ho, C. M., Guan, S., Mok, P. K., Lam, C. H., Ho, W. Y., Mak, C. H., Qin, H., Wong, A. K. C., & Hui, V. (2025). Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education. Journal of Medical Internet Research, 27, e74299. https://doi.org/10.2196/74299

Vancouver

Ho CM, Guan S, Mok PK, Lam CH, Ho WY, Mak CH, et al. Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education. Journal of Medical Internet Research. 2025;27:e74299. doi:10.2196/74299

BibTeX
@article{Ho2025,
  author = {Ho, Chung Man and Guan, Shaowei and Mok, Prudence Kwan-Lam and Lam, Candice Hw and Ho, Wai Ying and Mak, Calvin Hoi-Kwan and Qin, Harry and Wong, Arkers Kwan Ching and Hui, Vivian},
  title = {Development and Validation of a Large Language Model–Powered Chatbot for Neurosurgery: Mixed Methods Study on Enhancing Perioperative Patient Education},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e74299},
  doi = {10.2196/74299}
}
Start for free →