EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections

M Bortoli, M Fiore, S Tedeschi, V Oliveira, R Sousa, A Bruschi, D A Campanacci, P Viale, M De Paolis, A Sambri

MUSCULOSKELETAL SURGERY · 2024-12 · 108 (4) · 459-466

DOI: 10.1007/s12306-024-00846-w ↗

Kurz gefasst

Fachartikel in der Zeitschrift MUSCULOSKELETAL SURGERY, online erschienen am 02.07.2024. Verfasst von M Bortoli und 9 weiteren Autor:innen. Beteiligte Einrichtung: Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy. Hauptthemen laut MEDLINE: Prosthesis-Related Infections. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Alma Mater Studiorum - Università di Bologna.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence chatbot tools responses might discern patterns and correlations that may elude human observation, leading to more accurate and timely interventions. However, their reliability to answer healthcare-related questions is still debated. This study aimed to assess the performance of the three versions of GPT-based chatbots about prosthetic joint infections (PJI).

Methods

Thirty questions concerning the diagnosis and treatment of hip and knee PJIs, stratified by a priori established difficulty, were generated by a team of experts, and administered to ChatGPT 3.5, BingChat, and ChatGPT 4.0. Responses were rated by three orthopedic surgeons and two infectious diseases physicians using a five-point Likert-like scale with numerical values to quantify the quality of responses. Inter-rater reliability was assessed by interclass correlation statistics.

Results

Responses averaged “good-to-very good” for all chatbots examined, both in diagnosis and treatment, with no significant differences according to the difficulty of the questions. However, BingChat ratings were significantly lower in the treatment setting ( p = 0.025), particularly in terms of accuracy ( p = 0.02) and completeness ( p = 0.004). Agreement in ratings among examiners appeared to be very poor.

Conclusions

On average, the quality of responses is rated positively by experts, but with ratings that frequently may vary widely. This currently suggests that AI chatbot tools are still unreliable in the management of PJI.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Prosthesis-Related InfectionsHumansReproducibility of ResultsArtificial IntelligenceHip ProsthesisKnee ProsthesisSurveys and QuestionnairesArthroplasty, Replacement, HipArthroplasty, Replacement, Knee

Stichwörter der Autor:innen

Artificial intelligenceChatgptOrthopedic infectionPeriprosthetic joint infection

Autor:innen und Institutionen

  • Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy.
  • Orthopedic and Traumatology Unit, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy. michele.fiore9@unibo.it.
  • Department of Medical and Surgical Sciences, Alma Mater Studiorum University of Bologna, 40138, Bologna, Italy. michele.fiore9@unibo.it.
  • Department of Medical and Surgical Sciences, Alma Mater Studiorum University of Bologna, 40138, Bologna, Italy.
  • Infectious Disease Unit, Department for Integrated Infectious Risk Management, IRCCS Azienda Ospedaliero-Universitaria Di Bologna, 40138, Bologna, Italy.
  • Department of Orthopedics, Centro Hospitalar Universitário de Santo António, 4099-001, Porto, Portugal.

Förderung

  • Alma Mater Studiorum - Università di Bologna

Zitieren

APA

Bortoli, M., Fiore, M., Tedeschi, S., Oliveira, V., Sousa, R., Bruschi, A., Campanacci, D. A., Viale, P., De Paolis, M., & Sambri, A. (2024). GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections. MUSCULOSKELETAL SURGERY, 108(4), 459-466. https://doi.org/10.1007/s12306-024-00846-w

Vancouver

Bortoli M, Fiore M, Tedeschi S, Oliveira V, Sousa R, Bruschi A, et al. GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections. MUSCULOSKELETAL SURGERY. 2024;108(4):459-466. doi:10.1007/s12306-024-00846-w

BibTeX
@article{Bortoli2024,
  author = {Bortoli, M and Fiore, M and Tedeschi, S and Oliveira, V and Sousa, R and Bruschi, A and Campanacci, D A and Viale, P and De Paolis, M and Sambri, A},
  title = {GPT-based chatbot tools are still unreliable in the management of prosthetic joint infections},
  journal = {MUSCULOSKELETAL SURGERY},
  year = {2024},
  volume = {108},
  number = {4},
  pages = {459--466},
  doi = {10.1007/s12306-024-00846-w}
}
Kostenlos starten →