GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections
MUSCULOSKELETAL SURGERY · 2024-12 · 108 (4) · 459-466
DOI: 10.1007/s12306-024-00846-w ↗Kurz gefasst
Fachartikel in der Zeitschrift MUSCULOSKELETAL SURGERY, online erschienen am 02.07.2024. Verfasst von M Bortoli und 9 weiteren Autor:innen. Beteiligte Einrichtung: Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy. Hauptthemen laut MEDLINE: Prosthesis-Related Infections. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Alma Mater Studiorum - Università di Bologna.
Abstract (Originalsprache)
Background
Artificial intelligence chatbot tools responses might discern patterns and correlations that may elude human observation, leading to more accurate and timely interventions. However, their reliability to answer healthcare-related questions is still debated. This study aimed to assess the performance of the three versions of GPT-based chatbots about prosthetic joint infections (PJI).
Methods
Thirty questions concerning the diagnosis and treatment of hip and knee PJIs, stratified by a priori established difficulty, were generated by a team of experts, and administered to ChatGPT 3.5, BingChat, and ChatGPT 4.0. Responses were rated by three orthopedic surgeons and two infectious diseases physicians using a five-point Likert-like scale with numerical values to quantify the quality of responses. Inter-rater reliability was assessed by interclass correlation statistics.
Results
Responses averaged “good-to-very good” for all chatbots examined, both in diagnosis and treatment, with no significant differences according to the difficulty of the questions. However, BingChat ratings were significantly lower in the treatment setting ( p = 0.025), particularly in terms of accuracy ( p = 0.02) and completeness ( p = 0.004). Agreement in ratings among examiners appeared to be very poor.
Conclusions
On average, the quality of responses is rated positively by experts, but with ratings that frequently may vary widely. This currently suggests that AI chatbot tools are still unreliable in the management of PJI.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy.
- Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy. michele.fiore9@unibo.it.
- Department of Medical and Surgical Sciences, Alma Mater Studiorum University of Bologna, 40138, Bologna, Italy. michele.fiore9@unibo.it.
- Department of Medical and Surgical Sciences, Alma Mater Studiorum University of Bologna, 40138, Bologna, Italy.
- Infectious Disease Unit, Department for Integrated Infectious Risk Management, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy.
- Department of Orthopedics, Centro Hospitalar Universitário de Santo António, 4099-001, Porto, Portugal.
Förderung
- Alma Mater Studiorum - Università di Bologna
Zitieren
APA
Bortoli, M., Fiore, M., Tedeschi, S., Oliveira, V., Sousa, R., Bruschi, A., Campanacci, D. A., Viale, P., De Paolis, M., & Sambri, A. (2024). GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections. MUSCULOSKELETAL SURGERY, 108(4), 459-466. https://doi.org/10.1007/s12306-024-00846-w
Vancouver
Bortoli M, Fiore M, Tedeschi S, Oliveira V, Sousa R, Bruschi A, et al. GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections. MUSCULOSKELETAL SURGERY. 2024;108(4):459-466. doi:10.1007/s12306-024-00846-w
BibTeX
@article{Bortoli2024,
author = {Bortoli, M and Fiore, M and Tedeschi, S and Oliveira, V and Sousa, R and Bruschi, A and Campanacci, D A and Viale, P and De Paolis, M and Sambri, A},
title = {GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections},
journal = {MUSCULOSKELETAL SURGERY},
year = {2024},
volume = {108},
number = {4},
pages = {459--466},
doi = {10.1007/s12306-024-00846-w}
}
EUVIMED