EUVIMEDEUVIMED entdecken
Open Access

LMOD \(\boldsymbol{+}\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology

Zhenyue Qin iD, Yang Liu iD, Y U Yin iD, Jinyu Ding iD, Haoran Zhang iD, Anran Li iD, Dylan Campbell iD, Xuansheng Wu iD, K E Zou iD, Tiarnan D L Keenan iD, Emily Y Chew iD, Zhiyong Lu iD und 4 weitere

ACM Transactions on Computing for Healthcare · 30.07.2026 · 7 (3) · 1-38

DOI: 10.1145/3801746 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Computing for Healthcare, online erschienen am 22.06.2026. Verfasst von Zhenyue Qin und 15 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, Yale University, New Haven, CT, USA. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 85 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, National Library of Medicine, National Institutes of Health, National Library of Medicine and the National Eye Institute.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rising prevalence of vision-threatening eye diseases poses a major global health and economic burden, yet timely diagnosis remains limited by workforce shortages, diagnostic delays, and restricted access to specialized care. Artificial intelligence (AI) offers potential solutions. In particular, recent progress in foundation models and large language models—especially multimodal large language models (MLLMs)—has shown promise in medical image interpretation and automated clinical documentation. However, advancing MLLMs for ophthalmology is hindered by the lack of unified, comprehensive benchmark datasets for development and evaluation. Most existing benchmarks were designed for earlier models, which focused on narrow tasks or specific disease conditions. These benchmarks typically provide outputs in the form of disease labels rather than free-text responses. As a result, they are less suitable for assessing emerging generative models. In this work, we present LMOD+, a large-scale multimodal ophthalmology benchmark dataset comprising 32,633 instances with multi-granular annotations across 12 common ophthalmic conditions and 5 imaging modalities. The dataset integrates imaging, anatomical structures, demographics, and free-text annotations. It supports primary ophthalmic applications such as anatomical structure recognition, disease screening, disease staging, and demographic prediction for potential performance bias evaluation. Alongside the dataset, we introduce a systematic and unified data curation pipeline that repurposes existing or new datasets for MLLM development. LMOD+ extends our preliminary LMOD benchmark—the first multimodal ophthalmology benchmark for MLLMs—with three major enhancements. First, we expanded the dataset by nearly 50% (from 21,933 to 32,633 instances). The color fundus photography (CFP) modality, the most accessible imaging modality in ophthalmology, was significantly enlarged to cover a broader range of pathological conditions. Second, we broadened task coverage to include (a) 12 binary disease diagnosis tasks for prevalent conditions such as diabetic retinopathy, age-related macular degeneration, and retinal vein occlusion; (b) multi-class ophthalmic disease diagnosis; (c) disease severity classification, including a diabetic retinopathy staging task, which uses two internationally adopted grading standards: the international clinical diabetic retinopathy classification and the Scottish diabetic retinopathy grading scheme classification; and (d) demographic prediction (age and sex) to assess potential model bias. Third, we systematically evaluated 24 state-of-the-art MLLMs, including recent models from the InternVL, Qwen, and DeepSeek families. Our evaluations highlight both the promise and limitations of current MLLMs in ophthalmology. For example, Qwen-7B and InternVL achieved accuracies of 58.26% and 57.83% in disease screening under a zero-shot setting with a single model—a considerably more challenging paradigm than traditional fine-tuning, where separate models are trained for each specific task. InternVL also demonstrated potential in anatomical recognition. Nonetheless, overall performance remained suboptimal and often close to random baselines for challenging tasks such as disease staging, underscoring the substantial gap between general-domain MLLMs and the specialized requirements of ophthalmology. We publicly release the dataset, curation pipeline, and leaderboard to encourage community-wide development and evaluation of MLLMs, with the goal of advancing ophthalmic applications and ultimately reducing the global burden of vision-threatening diseases through AI. The dataset website, benchmark leaderboard, and download link are available at https://kfzyqin.github.io/lmod_plus .

Schlagwörter

Stichwörter der Autor:innen

Multimodal large language modelsbenchmark datasethealthcare computingmedical AIophthalmology

Autor:innen und Institutionen

  • School of Medicine, Yale University, New Haven, CT, USA.
  • School of Computing, Australian National University, Canberra, ACT, Australia.
  • School of Engineering, Imperial College London, London, UK.
  • School of Computing, University of Georgia, Athens, GA, USA.
  • Yong Loo Lin School of Medicine, National University of Singapore, Singapore.
  • National Eye Institute, National Institutes of Health, Bethesda, MD, USA.

ORCID-Profile: Zhenyue Qin, Yang Liu, Y U Yin, Jinyu Ding, Haoran Zhang, Anran Li, Dylan Campbell, Xuansheng Wu, K E Zou, Tiarnan D L Keenan, Emily Y Chew, Zhiyong Lu, Yih Chung Tham, Ninghao Liu, Xiuzhen Zhang, Qingyu Chen

Förderung

  • NLM NIH HHS
  • National Library of Medicine, National Institutes of Health
  • National Library of Medicine and the National Eye Institute
  • National Institutes of Health

Zitieren

APA

Qin, Z., Liu, Y., Yin, Y. U., Ding, J., Zhang, H., Li, A., Campbell, D., Wu, X., Zou, K. E., Keenan, T. D. L., Chew, E. Y., Lu, Z., Tham, Y. C., Liu, N., Zhang, X., & Chen, Q. (2026). LMOD \(\boldsymbol{+}\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology. ACM Transactions on Computing for Healthcare, 7(3), 1-38. https://doi.org/10.1145/3801746

Vancouver

Qin Z, Liu Y, Yin YU, Ding J, Zhang H, Li A, et al. LMOD \(\boldsymbol{+}\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology. ACM Transactions on Computing for Healthcare. 2026;7(3):1-38. doi:10.1145/3801746

BibTeX
@article{Qin2026,
  author = {Qin, Zhenyue and Liu, Yang and Yin, Y U and Ding, Jinyu and Zhang, Haoran and Li, Anran and Campbell, Dylan and Wu, Xuansheng and Zou, K E and Keenan, Tiarnan D L and Chew, Emily Y and Lu, Zhiyong and Tham, Yih Chung and Liu, Ninghao and Zhang, Xiuzhen and Chen, Qingyu},
  title = {LMOD \(\boldsymbol+\) : A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology},
  journal = {ACM Transactions on Computing for Healthcare},
  year = {2026},
  volume = {7},
  number = {3},
  pages = {1--38},
  doi = {10.1145/3801746}
}
Kostenlos starten →