EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity

Ali Hakim Reyhan iD, Çağrı Mutaf, İrfan Uzun iD, Funda Yüksekyayla iD

Journal of Clinical Medicine · 30.10.2024 · 13 (21) · 6512

DOI: 10.3390/jcm13216512 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Clinical Medicine, online erschienen am 30.10.2024. Verfasst von Ali Hakim Reyhan und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 Şanlıurfa, Türkiye. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

This study evaluates the ability of six popular chatbots; ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity, to provide reliable answers to questions concerning keratoconus.

Methods

Chatbots responses were assessed using mDISCERN (range: 15–75) and Global Quality Score (GQS) (range: 1–5) metrics. Readability was evaluated using nine validated readability assessments. We also addressed the quality and accountability of websites from which the questions originated.

Results

We analyzed 20 websites, 65% “Private practice or independent user” and 35% “Official patient education materials”. The mean JAMA benchmark score was 1.40 ± 0.91 (0–4 points), indicating low accountability. Reliability, measured using mDISCERN, ranged from 42.91 ± 3.15 (ChatGPT-3.5) to 46.95 ± 3.53 (Copilot). The most frequent question was “What is keratoconus?” with 70% of websites providing relevant information. This received the highest mDISCERN score (49.30 ± 4.91) and a relatively high GQS score (3.40 ± 0.56) with an Automated Readability Level Calculator score of 13.17 ± 2.13. Moderate positive correlations were determined between the website numbers and both mDISCERN (r = 0.265, p = 0.25) and GQS (r = 0.453, p = 0.05) scores. The quality of information, assessed using the GQS, ranged from 3.02 ± 0.55 (ChatGPT-3.5) to 3.31 ± 0.64 (Gemini) (p = 0.34). The differences between the texts were statistically significant. Gemini emerged as the easiest to read, while ChatGPT-3.5 and Perplexity were the most difficult. Based on mDISCERN scores, Gemini and Copilot exhibited the highest percentage of responses in the “good” range (51–62 points). For the GQS, the Gemini model exhibited the highest percentage of responses in the “good” quality range with 40% of its responses scoring 4–5.

Conclusions

While all chatbots performed well, Gemini and Copilot showed better reliability and quality. However, their readability often exceeded recommended levels. Continuous improvements are essential to match information with patients’ health literacy for effective use in ophthalmology.

Schlagwörter

Stichwörter der Autor:innen

chatbotskeratoconuslarge language models

Autor:innen und Institutionen

  • Department of Ophthalmology, Faculty of Medicine, Harran University, 63100 Şanlıurfa, Türkiye.

ORCID-Profile: Ali Hakim Reyhan, İrfan Uzun, Funda Yüksekyayla

Zitieren

APA

Reyhan, A. H., Mutaf, Ç., Uzun, İ., & Yüksekyayla, F. (2024). A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity. Journal of Clinical Medicine, 13(21), 6512. https://doi.org/10.3390/jcm13216512

Vancouver

Reyhan AH, Mutaf Ç, Uzun İ, Yüksekyayla F. A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity. Journal of Clinical Medicine. 2024;13(21):6512. doi:10.3390/jcm13216512

BibTeX
@article{Reyhan2024,
  author = {Reyhan, Ali Hakim and Mutaf, Çağrı and Uzun, İrfan and Yüksekyayla, Funda},
  title = {A Performance Evaluation of Large Language Models in Keratoconus: A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity},
  journal = {Journal of Clinical Medicine},
  year = {2024},
  volume = {13},
  number = {21},
  pages = {6512},
  doi = {10.3390/jcm13216512}
}
Kostenlos starten →