EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access

Evaluating Large Language Models in Ophthalmology: Systematic Review

Zili Zhang iD, Haiyang Zhang iD, Zhe Pan iD, Zhangqian Bi iD, Yao Wan iD, Xuefei Song iD, Xianqun Fan iD

Journal of Medical Internet Research · 27.10.2025 · 27 · e76947

DOI: 10.2196/76947 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Journal of Medical Internet Research, online erschienen am 27.10.2025. Verfasst von Zili Zhang und 6 weiteren Autor:innen. Beteiligte Einrichtung: State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, …. Hauptthemen laut MEDLINE: Ophthalmology, Language. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 239 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration.

Objective

This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task.

Methods

A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty.

Results

Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4).

Conclusions

Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

OphthalmologyLanguageHumansLarge Language Models

Stichwörter der Autor:innen

artificial intelligenceclinical evaluationlarge language modelmeta-analysisophthalmologysystematic review

Autor:innen und Institutionen

  • State Key Laboratory of Eye Health, Department of Ophthalmology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
  • School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.

ORCID-Profile: Zili Zhang, Haiyang Zhang, Zhe Pan, Zhangqian Bi, Yao Wan, Xuefei Song, Xianqun Fan

Zitieren

APA

Zhang, Z., Zhang, H., Pan, Z., Bi, Z., Wan, Y., Song, X., & Fan, X. (2025). Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research, 27, e76947. https://doi.org/10.2196/76947

Vancouver

Zhang Z, Zhang H, Pan Z, Bi Z, Wan Y, Song X, et al. Evaluating Large Language Models in Ophthalmology: Systematic Review. Journal of Medical Internet Research. 2025;27:e76947. doi:10.2196/76947

BibTeX
@article{Zhang2025,
  author = {Zhang, Zili and Zhang, Haiyang and Pan, Zhe and Bi, Zhangqian and Wan, Yao and Song, Xuefei and Fan, Xianqun},
  title = {Evaluating Large Language Models in Ophthalmology: Systematic Review},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e76947},
  doi = {10.2196/76947}
}
Kostenlos starten →