Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma
Journal of Surgical Oncology · 2025-03 · 131 (4) · 719-724
DOI: 10.1002/jso.27966 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Surgical Oncology, online erschienen am 29.10.2024. Verfasst von Kameel Khabaz und 6 weiteren Autor:innen. Beteiligte Einrichtung: David Geffen School of Medicine at UCLA, Los Angeles, California, USA. Hauptthemen laut MEDLINE: Artificial Intelligence, Bone Neoplasms, Patient Education as Topic, Osteosarcoma. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NIAMS NIH HHS, The authors received no specific funding for this work..
Abstract (Originalsprache)
Background and Objectives The potential impacts of artificial intelligence (AI) chatbots on care for patients with bone sarcoma is poorly understood. Elucidating potential risks and benefits would allow surgeons to define appropriate roles for these tools in clinical care.
Methods
Eleven questions on bone sarcoma diagnosis, treatment, and recovery were inputted into three AI chatbots. Answers were assessed on a 5‐point Likert scale for five clinical accuracy metrics: relevance to the question, balance and lack of bias, basis on established data, factual accuracy, and completeness in scope. Responses were quantitatively assessed for empathy and readability. The Patient Education Materials Assessment Tool (PEMAT) was assessed for understandability and actionability.
Results
Chatbots scored highly on relevance (4.24) and balance/lack of bias (4.09) but lower on basing responses on established data (3.77), completeness (3.68), and factual accuracy (3.66). Responses generally scored well on understandability (84.30%), while actionability scores were low for questions on treatment (64.58%) and recovery (60.64%). GPT‐4 exhibited the highest empathy (4.12). Readability scores averaged between 10.28 for diagnosis questions to 11.65 for recovery questions.
Conclusions
While AI chatbots are promising tools, current limitations in factual accuracy and completeness, as well as concerns of inaccessibility to populations with lower health literacy, may significantly limit their clinical utility.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- David Geffen School of Medicine at UCLA, Los Angeles, California, USA.
- Department of Orthopaedic Surgery, University of California, Los Angeles, California, USA.
- Department of Surgery, University of Calgary, Calgary, Alberta, Canada.
ORCID-Profile: Lauren E Wessel
Förderung
- NIAMS NIH HHS
- The authors received no specific funding for this work.
Zitieren
APA
Khabaz, K., Newman-Hung, N. J., Kallini, J. R., Kendal, J., Christ, A. B., Bernthal, N. M., & Wessel, L. E. (2025). Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma. Journal of Surgical Oncology, 131(4), 719-724. https://doi.org/10.1002/jso.27966
Vancouver
Khabaz K, Newman-Hung NJ, Kallini JR, Kendal J, Christ AB, Bernthal NM, et al. Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma. Journal of Surgical Oncology. 2025;131(4):719-724. doi:10.1002/jso.27966
BibTeX
@article{Khabaz2025,
author = {Khabaz, Kameel and Newman-Hung, Nicole J and Kallini, Jennifer R and Kendal, Joseph and Christ, Alexander B and Bernthal, Nicholas M and Wessel, Lauren E},
title = {Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma},
journal = {Journal of Surgical Oncology},
year = {2025},
volume = {131},
number = {4},
pages = {719--724},
doi = {10.1002/jso.27966}
}
EUVIMED