EUVIMEDOpdag EUVIMED
VergleichsstudieOpen access

Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study

Xuexue Bai iD, Shiyong Wang iD, Yuanli Zhao iD, Ming Feng iD, Wenbin Ma iD, Xiaomin Liu iD

Journal of Medical Internet Research · 2025-05-21 · 27 · e67462

DOI: 10.2196/67462 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2025-05-21. Written by Xuexue Bai and 5 co-authors. Institution: Department of Neurosurgery, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China. Major MEDLINE topics: Neoplasms, Artificial Intelligence, Text Messaging. Cited 17 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Telemedicine, which incorporates artificial intelligence such as chatbots, offers significant potential for enhancing health care delivery. However, the efficacy of artificial intelligence chatbots compared to human physicians in clinical settings remains underexplored, particularly in complex scenarios involving patients with cancer and asynchronous text-based interactions.

Objective

This study aimed to evaluate the performance of the GPT-4 (OpenAI) chatbot in responding to asynchronous text-based medical messages from patients with cancer by comparing its responses with those of physicians across two clinical scenarios: patient education and medical decision-making.

Methods

We collected 4257 deidentified asynchronous text-based medical consultation records from 17 oncologists across China between January 1, 2020, and March 31, 2024. Each record included patient questions, demographic data, and disease-related details. The records were categorized into two scenarios: patient education (eg, symptom explanations and test interpretations) and medical decision-making (eg, treatment planning). The GPT-4 chatbot was used to simulate physician responses to these records, with each session conducted in a new conversation to avoid cross-session interference. The chatbot responses, along with the original physician responses, were evaluated by a medical review panel (3 oncologists) and a patient panel (20 patients with cancer). The medical panel assessed completeness, accuracy, and safety using a 3-level scale, whereas the patient panel rated completeness, trustworthiness, and empathy on a 5-point ordinal scale. Statistical analyses included chi-square tests for categorical variables and Wilcoxon signed-rank tests for ordinal ratings.

Results

In the patient education scenario (n=2364), the chatbot scored higher than physicians in completeness (n=2301, 97.34% vs n=2213, 93.61% for fully complete responses; P=.002), with no significant differences in accuracy or safety (P>.05). In the medical decision-making scenario (n=1893), the chatbot exhibited lower accuracy (n=1834, 96.88% vs n=1855, 97.99% for fully accurate responses; P

Subjects

MeSH terms (MEDLINE)

NeoplasmsArtificial IntelligenceText MessagingHumansMaleFemaleMiddle AgedTelemedicinePatient Education as TopicChinaAdultPhysician-Patient RelationsAgedGenerative Artificial Intelligence

Author keywords

artificial intelligence, chatbot, telemedicinecomparative studyoncology

Authors and institutions

  • Department of Neurosurgery, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China.
  • Department of Neurosurgery, Peking Union Medical College Hospital, Beijing, China.
  • Department of Neurosurgery, First Affiliated Hospital of Jinan University, Guangzhou, China.
  • Head and Neck Neuro-Oncology Center, Tianjin Huanhu Hospital, Tianjin, China.

ORCID profiles: Xuexue Bai, Shiyong Wang, Yuanli Zhao, Ming Feng, Wenbin Ma, Xiaomin Liu

Cite

APA

Bai, X., Wang, S., Zhao, Y., Feng, M., Ma, W., & Liu, X. (2025). Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study. Journal of Medical Internet Research, 27, e67462. https://doi.org/10.2196/67462

Vancouver

Bai X, Wang S, Zhao Y, Feng M, Ma W, Liu X. Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study. Journal of Medical Internet Research. 2025;27:e67462. doi:10.2196/67462

BibTeX
@article{Bai2025,
  author = {Bai, Xuexue and Wang, Shiyong and Zhao, Yuanli and Feng, Ming and Ma, Wenbin and Liu, Xiaomin},
  title = {Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e67462},
  doi = {10.2196/67462}
}
Start for free →