EUVIMEDEUVIMED entdecken
Crossref

Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication

Mehmet Okan Atahan iD, Çağrı Üner iD, Mehmet Aydemir iD, Mehmet Fatih Uzun iD, Mustafa Yalın iD, Fatih Gölgelioğlu iD

Journal of Evaluation in Clinical Practice · 2025

DOI: 10.1111/jep.70238 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2025. Verfasst von Mehmet Okan Atahan und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopedics and Traumatology, Afyonkarahisar State Hospital, Afyonkarahisar, Turkey. Hauptthemen laut MEDLINE: Comprehension, Orthopedics, Communication, Wounds and Injuries. Die Arbeit nennt 29 Literaturangaben. Gefördert von: The authors received no specific funding for this work..

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aim

This study aimed to evaluate the accuracy, readability, and safety of ChatGPT‐4.0's responses to frequently asked questions (FAQs) related to orthopaedic trauma and to examine whether readability is associated with the quality and reliability of content.

Methods

Ten common patient questions related to orthopaedic emergencies were submitted to ChatGPT‐4.0. Each response was assessed independently by three orthopaedic trauma surgeons using a 4‐point ordinal scale for accuracy, clinical appropriateness, and safety. Readability was calculated using the Flesch‐Kincaid Grade Level (FKGL). Inter‐rater agreement was analysed using intraclass correlation coefficients (ICC). The presence of disclaimers was also recorded.

Results

ChatGPT‐4.0's responses had a mean FKGL score of 10.5, indicating high school‐level readability. Stratified analysis showed comparable readability scores across response quality categories: excellent (10.0), poor (9.8), and dangerous (10.1), suggesting that readability does not predict content reliability. Accuracy and safety scores varied considerably among responses, with the highest inter‐rater agreement in clinical appropriateness (ICC = 0.81) and the lowest in safety assessments (ICC = 0.68). Notably, nine out of 10 responses included a disclaimer indicating the nonprofessional nature of the content, with one omission observed in a high‐risk clinical scenario.

Conclusion

Although ChatGPT‐4.0 provides generally readable responses to orthopaedic trauma questions, readability does not reliably distinguish between accurate and potentially harmful information. These findings highlight the need for expert review when using AI‐generated content in clinical communication.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ComprehensionOrthopedicsCommunicationWounds and InjuriesHumansReproducibility of ResultsSurveys and QuestionnairesQualitative ResearchGenerative Artificial Intelligence

Stichwörter der Autor:innen

artificial intelligencehealth educationnatural language processingorthopaedic proceduresreadability

Autor:innen und Institutionen

  • Department of Orthopedics and Traumatology, Afyonkarahisar State Hospital, Afyonkarahisar, Turkey.
  • Department of Orthopedics and Traumatology, Yozgat City Hospital, Yozgat, Turkey.
  • Department of Orthopedics and Traumatology, Ceylanpınar State Hospital, Şanlıurfa, Turkey.
  • Department of Orthopedics and Traumatology, Yalova University Faculty of Medicine, Yalova, Turkey.
  • Department of Orthopedics and Traumatology, Bozok University Faculty of Medicine, Yozgat, Turkey.

ORCID-Profile: Mehmet Okan Atahan, Çağrı Üner, Mehmet Aydemir, Mehmet Fatih Uzun, Mustafa Yalın, Fatih Gölgelioğlu

Förderung

  • The authors received no specific funding for this work.

Zitieren

APA

Atahan, M. O., Üner, Ç., Aydemir, M., Uzun, M. F., Yalın, M., & Gölgelioğlu, F. (2025). Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70238

Vancouver

Atahan MO, Üner Ç, Aydemir M, Uzun MF, Yalın M, Gölgelioğlu F. Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication. Journal of Evaluation in Clinical Practice. 2025. doi:10.1111/jep.70238

BibTeX
@article{Atahan2025,
  author = {Atahan, Mehmet Okan and Üner, Çağrı and Aydemir, Mehmet and Uzun, Mehmet Fatih and Yalın, Mustafa and Gölgelioğlu, Fatih},
  title = {Is Readability a Proxy for Reliability? A Qualitative Evaluation of ChatGPT‐4.0 in Orthopaedic Trauma Communication},
  journal = {Journal of Evaluation in Clinical Practice},
  year = {2025},
  doi = {10.1111/jep.70238}
}
Kostenlos starten →