Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study
Journal of Medical Internet Research · 21.05.2025 · 27 · e67462
DOI: 10.2196/67462 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 21.05.2025. Verfasst von Xuexue Bai und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurosurgery, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China. Hauptthemen laut MEDLINE: Neoplasms, Artificial Intelligence, Text Messaging. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Telemedicine, which incorporates artificial intelligence such as chatbots, offers significant potential for enhancing health care delivery. However, the efficacy of artificial intelligence chatbots compared to human physicians in clinical settings remains underexplored, particularly in complex scenarios involving patients with cancer and asynchronous text-based interactions.
Objective
This study aimed to evaluate the performance of the GPT-4 (OpenAI) chatbot in responding to asynchronous text-based medical messages from patients with cancer by comparing its responses with those of physicians across two clinical scenarios: patient education and medical decision-making.
Methods
We collected 4257 deidentified asynchronous text-based medical consultation records from 17 oncologists across China between January 1, 2020, and March 31, 2024. Each record included patient questions, demographic data, and disease-related details. The records were categorized into two scenarios: patient education (eg, symptom explanations and test interpretations) and medical decision-making (eg, treatment planning). The GPT-4 chatbot was used to simulate physician responses to these records, with each session conducted in a new conversation to avoid cross-session interference. The chatbot responses, along with the original physician responses, were evaluated by a medical review panel (3 oncologists) and a patient panel (20 patients with cancer). The medical panel assessed completeness, accuracy, and safety using a 3-level scale, whereas the patient panel rated completeness, trustworthiness, and empathy on a 5-point ordinal scale. Statistical analyses included chi-square tests for categorical variables and Wilcoxon signed-rank tests for ordinal ratings.
Results
In the patient education scenario (n=2364), the chatbot scored higher than physicians in completeness (n=2301, 97.34% vs n=2213, 93.61% for fully complete responses; P=.002), with no significant differences in accuracy or safety (P>.05). In the medical decision-making scenario (n=1893), the chatbot exhibited lower accuracy (n=1834, 96.88% vs n=1855, 97.99% for fully accurate responses; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Neurosurgery, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China.
- Department of Neurosurgery, Peking Union Medical College Hospital, Beijing, China.
- Department of Neurosurgery, First Affiliated Hospital of Jinan University, Guangzhou, China.
- Head and Neck Neuro-Oncology Center, Tianjin Huanhu Hospital, Tianjin, China.
ORCID-Profile: Xuexue Bai, Shiyong Wang, Yuanli Zhao, Ming Feng, Wenbin Ma, Xiaomin Liu
Zitieren
APA
Bai, X., Wang, S., Zhao, Y., Feng, M., Ma, W., & Liu, X. (2025). Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study. Journal of Medical Internet Research, 27, e67462. https://doi.org/10.2196/67462
Vancouver
Bai X, Wang S, Zhao Y, Feng M, Ma W, Liu X. Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study. Journal of Medical Internet Research. 2025;27:e67462. doi:10.2196/67462
BibTeX
@article{Bai2025,
author = {Bai, Xuexue and Wang, Shiyong and Zhao, Yuanli and Feng, Ming and Ma, Wenbin and Liu, Xiaomin},
title = {Application of AI Chatbot in Responding to Asynchronous Text-Based Messages From Patients With Cancer: Comparative Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e67462},
doi = {10.2196/67462}
}
EUVIMED