Evaluating Large Language Models in Ophthalmology: Systematic Review
Journal of Medical Internet Research · 2025-10-27 · 27 · e76947
DOI: 10.2196/76947 ↗At a glance
Systematische Übersicht in Journal of Medical Internet Research, published online on 2025-10-27. Written by Zili Zhang and 6 co-authors. Institution: State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, …. Major MEDLINE topics: Ophthalmology, Language. Cited 41 times so far (Crossref); 239 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration.
Objective
This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task.
Methods
A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty.
Results
Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4).
Conclusions
Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
ORCID profiles: Zili Zhang, Haiyang Zhang, Zhe Pan, Zhangqian Bi, Yao Wan, Xuefei Song, Xianqun Fan
Cite
APA
Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025). Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research, 27, e76947. https://doi.org/10.2196/76947
Vancouver
Zhang Z, Zhang H, Pan Z, Bi Z, Wan Y, Song X, et al. Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research. 2025;27:e76947. doi:10.2196/76947
BibTeX
@article{Zhang2025,
author = {Zhang, Zili and Zhang, Haiyang and Pan, Zhe and Bi, Zhangqian and Wan, Yao and Song, Xuefei and Fan, Xianqun},
title = {Evaluating Large Language Models in Ophthalmology: Systematic Review},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e76947},
doi = {10.2196/76947}
}
EUVIMED