Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study
Journal of Medical Internet Research · 07.10.2026 · 28 · e86726-e86726
DOI: 10.2196/86726 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 07.10.2026. Verfasst von Shaojuan Peng und 10 weiteren Autor:innen. Beteiligte Einrichtung: Jinan University, Guangzhou, China. Hauptthemen laut MEDLINE: Retinopathy of Prematurity, Artificial Intelligence. Die Arbeit nennt 57 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Retinopathy of Prematurity (ROP) is a leading cause of preventable childhood blindness; yet, a global shortage of experienced pediatric ophthalmologists impedes timely diagnosis and treatment. While emerging AI chatbots are promising clinical decision-support tools in some ophthalmic diseases, their performance in ROP diagnosis and providing treatment suggestions remains uncertain.
Objective
This study aimed to compare the performance of Google’s Gemini 2.5 Pro and OpenAI’s ChatGPT o4-mini in ROP diagnosis and providing treatment suggestions against the gold standard of clinical consensus.
Methods
A retrospective analysis was conducted on 70 infants (140 eyes) with treatment-requiring ROP, each providing structured clinical text data and wide-field fundus images. We adopted a 2-stage prompting strategy for AI chatbots, instructing them first to generate ROP diagnoses (including zone, stage, and presence of plus disease), and subsequently to provide treatment suggestions. After collecting the generated responses, we assessed their performance by comparing the consistency of their diagnosis and treatment suggestions with the consensus gold standard. Furthermore, 2 independent specialists quantitatively assessed the outputs of Gemini 2.5 Pro and ChatGPT o4-mini using the ROP-specific Global Quality Score (GQS), which is a 5-point scale ranging from 1 (unusable) to 5 (excellent). Statistical significance was set at P .05). However, Gemini 2.5 Pro showed significantly better performance in plus disease identification (consistency: 108/140, 77.1% vs 80/140, 57.1%; P =.006), while ChatGPT o4-mini demonstrated significantly higher guideline adherence in treatment modality suggestions based on gold-standard ROP diagnoses (consistency: 91/140, 65.0%; vs 55/140, 39.3%; P =.01). Compared to Gemini 2.5 Pro, ChatGPT o4-mini performed better in providing ROP treatment suggestions (GQS score; P =.001), while the 2 AI chatbots had comparable GQS scores in diagnostic tasks.
Conclusions
ChatGPT o4-mini shows greater promise in generating evidence-based treatment suggestions based on gold-standard diagnoses, whereas Gemini 2.5 Pro shows advantages in visual interpretation, supporting its potential for targeted ROP diagnostic screening, particularly in identifying plus disease. As these AI chatbots continue to evolve, their performance merits further validation using larger cohorts.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Jinan University, Guangzhou, China.
- Shenzhen Eye Medical Center, Shenzhen Eye Hospital, Southern Medical University, 18 Zetian Road, Futian District, Shenzhen, 518043, China, 1 0755-23959600.
- Huizhou Third People's Hospital, First Affiliated Hospital of Jinan University, Guangzhou, China.
- Beijing Tongren Eye Center, Capital Medical University, Beijing, China.
Zitieren
APA
Peng, S., Zhao, X., Wu, Z., Yuan, D., Duan, N., Cui, K., Yu, Z., Yang, W., Wei, W., Chi, W., & Zhang, G. (2026). Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study. Journal of Medical Internet Research, 28, e86726-e86726. https://doi.org/10.2196/86726
Vancouver
Peng S, Zhao X, Wu Z, Yuan D, Duan N, Cui K, et al. Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study. Journal of Medical Internet Research. 2026;28:e86726-e86726. doi:10.2196/86726
BibTeX
@article{Peng2026,
author = {Peng, Shaojuan and Zhao, Xinyu and Wu, Zhenquan and Yuan, Duo and Duan, Na and Cui, Kaixuan and Yu, Zhen and Yang, Weihua and Wei, Wenbin and Chi, Wei and Zhang, Guoming},
title = {Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e86726--e86726},
doi = {10.2196/86726}
}
EUVIMED