A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity
Journal of Clinical Medicine · 30.10.2024 · 13 (21) · 6512
DOI: 10.3390/jcm13216512 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 30.10.2024. Verfasst von Ali Hakim Reyhan und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 Şanlıurfa, Türkiye. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
This study evaluates the ability of six popular chatbots; ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity, to provide reliable answers to questions concerning keratoconus.
Methods
Chatbots responses were assessed using mDISCERN (range: 15–75) and Global Quality Score (GQS) (range: 1–5) metrics. Readability was evaluated using nine validated readability assessments. We also addressed the quality and accountability of websites from which the questions originated.
Results
We analyzed 20 websites, 65% “Private practice or independent user” and 35% “Official patient education materials”. The mean JAMA benchmark score was 1.40 ± 0.91 (0–4 points), indicating low accountability. Reliability, measured using mDISCERN, ranged from 42.91 ± 3.15 (ChatGPT-3.5) to 46.95 ± 3.53 (Copilot). The most frequent question was “What is keratoconus?” with 70% of websites providing relevant information. This received the highest mDISCERN score (49.30 ± 4.91) and a relatively high GQS score (3.40 ± 0.56) with an Automated Readability Level Calculator score of 13.17 ± 2.13. Moderate positive correlations were determined between the website numbers and both mDISCERN (r = 0.265, p = 0.25) and GQS (r = 0.453, p = 0.05) scores. The quality of information, assessed using the GQS, ranged from 3.02 ± 0.55 (ChatGPT-3.5) to 3.31 ± 0.64 (Gemini) (p = 0.34). The differences between the texts were statistically significant. Gemini emerged as the easiest to read, while ChatGPT-3.5 and Perplexity were the most difficult. Based on mDISCERN scores, Gemini and Copilot exhibited the highest percentage of responses in the “good” range (51–62 points). For the GQS, the Gemini model exhibited the highest percentage of responses in the “good” quality range with 40% of its responses scoring 4–5.
Conclusions
While all chatbots performed well, Gemini and Copilot showed better reliability and quality. However, their readability often exceeded recommended levels. Continuous improvements are essential to match information with patients’ health literacy for effective use in ophthalmology.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 Şanlıurfa, Türkiye.
ORCID-Profile: Ali Hakim Reyhan, İrfan Uzun, Funda Yüksekyayla
Zitieren
APA
Reyhan, A. H., Mutaf, Ç., Uzun, İ., & Yüksekyayla, F. (2024). A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity. Journal of Clinical Medicine, 13(21), 6512. https://doi.org/10.3390/jcm13216512
Vancouver
Reyhan AH, Mutaf Ç, Uzun İ, Yüksekyayla F. A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity. Journal of Clinical Medicine. 2024;13(21):6512. doi:10.3390/jcm13216512
BibTeX
@article{Reyhan2024,
author = {Reyhan, Ali Hakim and Mutaf, Çağrı and Uzun, İrfan and Yüksekyayla, Funda},
title = {A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity},
journal = {Journal of Clinical Medicine},
year = {2024},
volume = {13},
number = {21},
pages = {6512},
doi = {10.3390/jcm13216512}
}
EUVIMED