Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication
Journal of Evaluation in Clinical Practice · 2025
DOI: 10.1111/jep.70238 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2025. Verfasst von Mehmet Okan Atahan und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedics and Traumatology, Afyonkarahisar State Hospital, Afyonkarahisar, Turkey. Hauptthemen laut MEDLINE: Comprehension, Orthopedics, Communication, Wounds and Injuries. Die Arbeit nennt 29 Literaturangaben. Gefördert von: The authors received no specific funding for this work..
Abstract (Originalsprache)
Aim
This study aimed to evaluate the accuracy, readability, and safety of ChatGPT‐4.0's responses to frequently asked questions (FAQs) related to orthopaedic trauma and to examine whether readability is associated with the quality and reliability of content.
Methods
Ten common patient questions related to orthopaedic emergencies were submitted to ChatGPT‐4.0. Each response was assessed independently by three orthopaedic trauma surgeons using a 4‐point ordinal scale for accuracy, clinical appropriateness, and safety. Readability was calculated using the Flesch‐Kincaid Grade Level (FKGL). Inter‐rater agreement was analysed using intraclass correlation coefficients (ICC). The presence of disclaimers was also recorded.
Results
ChatGPT‐4.0's responses had a mean FKGL score of 10.5, indicating high school‐level readability. Stratified analysis showed comparable readability scores across response quality categories: excellent (10.0), poor (9.8), and dangerous (10.1), suggesting that readability does not predict content reliability. Accuracy and safety scores varied considerably among responses, with the highest inter‐rater agreement in clinical appropriateness (ICC = 0.81) and the lowest in safety assessments (ICC = 0.68). Notably, nine out of 10 responses included a disclaimer indicating the nonprofessional nature of the content, with one omission observed in a high‐risk clinical scenario.
Conclusion
Although ChatGPT‐4.0 provides generally readable responses to orthopaedic trauma questions, readability does not reliably distinguish between accurate and potentially harmful information. These findings highlight the need for expert review when using AI‐generated content in clinical communication.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthopedics and Traumatology, Afyonkarahisar State Hospital, Afyonkarahisar, Turkey.
- Department of Orthopedics and Traumatology, Yozgat City Hospital, Yozgat, Turkey.
- Department of Orthopedics and Traumatology, Ceylanpınar State Hospital, Şanlıurfa, Turkey.
- Department of Orthopedics and Traumatology, Yalova University Faculty of Medicine, Yalova, Turkey.
- Department of Orthopedics and Traumatology, Bozok University Faculty of Medicine, Yozgat, Turkey.
ORCID-Profile: Mehmet Okan Atahan, Çağrı Üner, Mehmet Aydemir, Mehmet Fatih Uzun, Mustafa Yalın, Fatih Gölgelioğlu
Förderung
- The authors received no specific funding for this work.
Zitieren
APA
Atahan, M. O., Üner, Ç., Aydemir, M., Uzun, M. F., Yalın, M., & Gölgelioğlu, F. (2025). Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70238
Vancouver
Atahan MO, Üner Ç, Aydemir M, Uzun MF, Yalın M, Gölgelioğlu F. Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication. Journal of Evaluation in Clinical Practice. 2025. doi:10.1111/jep.70238
BibTeX
@article{Atahan2025,
author = {Atahan, Mehmet Okan and Üner, Çağrı and Aydemir, Mehmet and Uzun, Mehmet Fatih and Yalın, Mustafa and Gölgelioğlu, Fatih},
title = {Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication},
journal = {Journal of Evaluation in Clinical Practice},
year = {2025},
doi = {10.1111/jep.70238}
}
EUVIMED