Performance of ChatGPT-4o, Gemini 2.0 Pro, and DeepSeek-V3 in Patient-Facing Information on Chest Wall Deformities: A Comparative Evaluation of Accuracy, RELIABILITY, and Reproducibility
Diagnostics · 15.02.2026 · 16 (4) · 589
DOI: 10.3390/diagnostics16040589 ↗Kurz gefasst
Fachartikel in der Zeitschrift Diagnostics, online erschienen am 15.02.2026. Verfasst von Deniz Oke und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Physical Medicine and Rehabilitation, Health Sciences University Gaziosmanpasa Training and Research Hospital, Istanbul 34255, Turkey. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Large language models (LLMs) such as DeepSeek-V3, Google Gemini 2.0 Pro, and ChatGPT-4o are increasingly used by patients seeking online medical information. However, their accuracy, reliability, and reproducibility in patient-facing content related to chest wall deformities (CWD) remain unclear. This study aimed to compare the performance of three contemporary LLMs in generating information on pectus excavatum, pectus carinatum, and related thoracic deformities.
Methods
Eighty patient-facing questions were developed across eight thematic domains and independently submitted to each model using newly created accounts over two consecutive days. Accuracy was assessed using a validated four-point rubric by blinded physiatrists, and reproducibility was evaluated using agreement metrics and weighted Cohen’s kappa.
Results
ChatGPT-4o achieved the highest overall accuracy (median score: 1.20), the greatest proportion of fully accurate responses, and the lowest hallucination rate (5.0%). Gemini showed intermediate accuracy, while DeepSeek-V3 demonstrated the lowest accuracy and highest hallucination rate (11.25%). Across all models, general-information and quality-of-life domains had the best performance, whereas treatment-related questions showed the most errors. Reproducibility was highest for ChatGPT-4o (weighted κ = almost perfect), followed by Gemini and DeepSeek-V3. Inter-rater reliability was substantial (Fleiss’ κ = 0.69).
Conclusions
Contemporary LLMs can generate largely accurate and reproducible patient-facing information on CWD, with ChatGPT-4o showing the strongest overall performance. This study provides the first domain-specific comparative evaluation of LLMs in CWD and integrates reproducibility analysis alongside accuracy and reliability assessment. While these tools may support patient education, treatment-related responses require caution, and LLMs should be used as adjuncts rather than substitutes for clinical counseling.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Physical Medicine and Rehabilitation, Health Sciences University Gaziosmanpasa Training and Research Hospital, Istanbul 34255, Turkey.
- Department of Physical Medicine and Rehabilitation, Fatih Sultan Mehmet Research and Training Hospital, Istanbul 34015, Turkey.
- Department of Physical Medicine and Rehabilitation, Kırklareli University Faculty of Medicine, Kırklareli 39060, Turkey.
ORCID-Profile: Deniz Oke, Betül Çiftçi
Zitieren
APA
Oke, D., Illeez, O. G., Giray, E., & Çiftçi, B. (2026). Performance of ChatGPT-4o, Gemini 2.0 Pro, and DeepSeek-V3 in Patient-Facing Information on Chest Wall Deformities: A Comparative Evaluation of Accuracy, RELIABILITY, and Reproducibility. Diagnostics, 16(4), 589. https://doi.org/10.3390/diagnostics16040589
Vancouver
Oke D, Illeez OG, Giray E, Çiftçi B. Performance of ChatGPT-4o, Gemini 2.0 Pro, and DeepSeek-V3 in Patient-Facing Information on Chest Wall Deformities: A Comparative Evaluation of Accuracy, RELIABILITY, and Reproducibility. Diagnostics. 2026;16(4):589. doi:10.3390/diagnostics16040589
BibTeX
@article{Oke2026,
author = {Oke, Deniz and Illeez, Ozge Gulsum and Giray, Esra and Çiftçi, Betül},
title = {Performance of ChatGPT-4o, Gemini 2.0 Pro, and DeepSeek-V3 in Patient-Facing Information on Chest Wall Deformities: A Comparative Evaluation of Accuracy, RELIABILITY, and Reproducibility},
journal = {Diagnostics},
year = {2026},
volume = {16},
number = {4},
pages = {589},
doi = {10.3390/diagnostics16040589}
}
EUVIMED