Evaluating Large Language Models in Ophthalmology: Systematic Review
Journal of Medical Internet Research · 27.10.2025 · 27 · e76947
DOI: 10.2196/76947 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Journal of Medical Internet Research, online erschienen am 27.10.2025. Verfasst von Zili Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, …. Hauptthemen laut MEDLINE: Ophthalmology, Language. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 239 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration.
Objective
This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task.
Methods
A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty.
Results
Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4).
Conclusions
Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
ORCID-Profile: Zili Zhang, Haiyang Zhang, Zhe Pan, Zhangqian Bi, Yao Wan, Xuefei Song, Xianqun Fan
Zitieren
APA
Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025). Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research, 27, e76947. https://doi.org/10.2196/76947
Vancouver
Zhang Z, Zhang H, Pan Z, Bi Z, Wan Y, Song X, et al. Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research. 2025;27:e76947. doi:10.2196/76947
BibTeX
@article{Zhang2025,
author = {Zhang, Zili and Zhang, Haiyang and Pan, Zhe and Bi, Zhangqian and Wan, Yao and Song, Xuefei and Fan, Xianqun},
title = {Evaluating Large Language Models in Ophthalmology: Systematic Review},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e76947},
doi = {10.2196/76947}
}
EUVIMED