EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Is the information provided by large language models valid in educating patients about adolescent idiopathic scoliosis? An evaluation of content, clarity, and empathy

Siegmund Lang, Jacopo Vitale, Fabio Galbusera iD, Tamás Fekete iD, Louis Boissiere, Yann Philippe Charles, Altug Yucekul, Caglar Yilgor, Susana Núñez-Pereira, Sleiman Haddad, Alejandro Gomez-Rice, Jwalant Mehta und 7 weitere

Spine Deformity · 2025

DOI: 10.1007/s43390-024-00955-3 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Spine Deformity, 2025. Verfasst von Siegmund Lang und 18 weiteren Autor:innen. Beteiligte Einrichtung: Department of Trauma Surgery, University Hospital Regensburg, Franz-Josef-Strauss-Allee 11, 93053, Regensburg, Germany. siegmund.lang@ukr.de. Hauptthemen laut MEDLINE: Scoliosis, Empathy, Patient Education as Topic. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universitätsklinikum Regensburg.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Large language models (LLM) have the potential to bridge knowledge gaps in patient education and enrich patient-surgeon interactions. This study evaluated three chatbots for delivering empathetic and precise adolescent idiopathic scoliosis (AIS) related information and management advice. Specifically, we assessed the accuracy, clarity, and relevance of the information provided, aiming to determine the effectiveness of LLMs in addressing common patient queries and enhancing their understanding of AIS.

Methods

We sourced 20 webpages for the top frequently asked questions (FAQs) about AIS and formulated 10 critical questions based on them. Three advanced LLMs—ChatGPT 3.5, ChatGPT 4.0, and Google Bard—were selected to answer these questions, with responses limited to 200 words. The LLMs’ responses were evaluated by a blinded group of experienced deformity surgeons (members of the European Spine Study Group) from seven European spine centers. A pre-established 4-level rating system from excellent to unsatisfactory was used with a further rating for clarity, comprehensiveness, and empathy on the 5-point Likert scale. If not rated 'excellent', the raters were asked to report the reasons for their decision for each question. Lastly, raters were asked for their opinion towards AI in healthcare in general in six questions.

Results

The responses among all LLMs were ‘excellent’ in 26% of responses, with ChatGPT-4.0 leading (39%), followed by Bard (17%). ChatGPT-4.0 was rated superior to Bard and ChatGPT 3.5 ( p = 0.003). Discrepancies among raters were significant ( p < 0.0001), questioning inter-rater reliability. No substantial differences were noted in answer distribution by question ( p = 0.43). The answers on diagnosis (Q2) and causes (Q4) of AIS were top-rated. The most dissatisfaction was seen in the answers regarding definitions (Q1) and long-term results (Q7). Exhaustiveness, clarity, empathy, and length of the answers were positively rated (> 3.0 on 5.0) and did not demonstrate any differences among LLMs. However, GPT-3.5 struggled with language suitability and empathy, while Bard’s responses were overly detailed and less empathetic. Overall, raters found that 9% of answers were off-topic and 22% contained clear mistakes.

Conclusion

Our study offers crucial insights into the strengths and weaknesses of current LLMs in AIS patient and parent education, highlighting the promise of advancements like ChatGPT-4.o and Gemini alongside the need for continuous improvement in empathy, contextual understanding, and language appropriateness .

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ScoliosisEmpathyPatient Education as TopicHumansAdolescentFemaleEuropeMalePhysician-Patient RelationsLarge Language Models

Stichwörter der Autor:innen

Adolescent idiopathic scoliosis (AIS)Artificial intelligence (AI)Large language models (LLMs)Patient educationSpine surgery

Autor:innen und Institutionen

  • Department of Trauma Surgery, University Hospital Regensburg, Franz-Josef-Strauss-Allee 11, 93053, Regensburg, Germany. siegmund.lang@ukr.de.
  • Department of Spine Surgery, Schulthess Klinik, Zurich, Switzerland. siegmund.lang@ukr.de.
  • Spine Center, Schulthess Klinik, Zurich, Switzerland.
  • Department of Spine Surgery, Schulthess Klinik, Zurich, Switzerland.
  • Spine Unit Orthopaedic Department, Hôpital Pellegrin Bordeaux, Bordeaux, France.
  • Dept. of Spine Surgery, Hôpitaux Universitaires de Strasbourg, Université de Strasbourg, Strasbourg, France.

ORCID-Profile: Fabio Galbusera, Tamás Fekete, Markus Loibl

Förderung

  • Universitätsklinikum Regensburg

Zitieren

APA

Lang, S., Vitale, J., Galbusera, F., Fekete, T., Boissiere, L., Charles, Y. P., Yucekul, A., Yilgor, C., Núñez-Pereira, S., Haddad, S., Gomez-Rice, A., Mehta, J., Pizones, J., Pellisé, F., Obeid, I., Alanay, A., Kleinstück, F., Loibl, M., & ESSG European Spine Study Group (2025). Is the information provided by large language models valid in educating patients about adolescent idiopathic scoliosis? An evaluation of content, clarity, and empathy. Spine Deformity. https://doi.org/10.1007/s43390-024-00955-3

Vancouver

Lang S, Vitale J, Galbusera F, Fekete T, Boissiere L, Charles YP, et al. Is the information provided by large language models valid in educating patients about adolescent idiopathic scoliosis? An evaluation of content, clarity, and empathy. Spine Deformity. 2025. doi:10.1007/s43390-024-00955-3

BibTeX
@article{Lang2025,
  author = {Lang, Siegmund and Vitale, Jacopo and Galbusera, Fabio and Fekete, Tamás and Boissiere, Louis and Charles, Yann Philippe and Yucekul, Altug and Yilgor, Caglar and Núñez-Pereira, Susana and Haddad, Sleiman and Gomez-Rice, Alejandro and Mehta, Jwalant and Pizones, Javier and Pellisé, Ferran and Obeid, Ibrahim and Alanay, Ahmet and Kleinstück, Frank and Loibl, Markus and ESSG European Spine Study Group, ESSG European Spine Study Group},
  title = {Is the information provided by large language models valid in educating patients about adolescent idiopathic scoliosis? An evaluation of content, clarity, and empathy},
  journal = {Spine Deformity},
  year = {2025},
  doi = {10.1007/s43390-024-00955-3}
}
Kostenlos starten →