EUVIMEDScopri EUVIMED
VergleichsstudieOpen access · CC BY 4.0

Evaluating the Impact of Retinal Vessel Segmentation Metrics on Retest Reliability in a Clinical Setting: A Comparative Analysis Using AutoMorph

Samuel D Giesser, Ferhat Turgut, Amr Saad, Jay R Zoellin, Chiara Sommer, Yukun Zhou, Siegfried K Wagner, Pearse A Keane, Matthias Becker, Delia Cabrera DeBuc, Gábor Márk Somfai

Investigative Ophthalmology & Visual Science · 2024-11-14 · 65 (13) · 24

DOI: 10.1167/iovs.65.13.24 ↗

At a glance

Vergleichsstudie in Investigative Ophthalmology & Visual Science, published on 2024-11-14. Written by Samuel D Giesser and 10 co-authors. Institution: Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland. Major MEDLINE topics: Retinal Vessels, Photography. Cited 10 times so far (Crossref); 34 references. The full text is freely available (open access, CC BY 4.0). Funded by: Medical Research Council.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Purpose

Current research on artificial intelligence-based fundus photography biomarkers has demonstrated inconsistent results. Consequently, we aimed to evaluate and predict the test-retest reliability of retinal parameters extracted from fundus photography.

Methods

Two groups of patients were recruited for the study: an intervisit group (n = 28) to assess retest reliability over a period of 1 to 14 days and an intravisit group (n = 44) to evaluate retest reliability within a single session. Using AutoMorph, we generated test and retest vessel segmentation maps; measured segmentation map agreement via accuracy, sensitivity, F1 score and Jaccard index; and calculated 76 metrics from each fundus image. The retest reliability of each metric was analyzed in terms of the Spearman correlation coefficient, intraclass correlation coefficient (ICC), and relative percentage change. A linear model with the input variables contrast-to-noise-ratio and fractal dimension, chosen by a P-value-based backward selection process, was developed to predict the median percentage difference on retest per image based on image-quality metrics. This model was trained on the intravisit dataset and validated using the intervisit dataset.

Results

In the intervisit group, retest reliability varied between Spearman correlation coefficients of 0.34 and 0.99, ICC values of 0.31 to 0.99, and mean absolute percentage differences of 0.96% to 223.67%. Similarly, in the intravisit group, the retest reliability ranged from Spearman correlation coefficients of 0.55 and 0.96, ICC values of 0.40 to 0.97, and mean percentage differences of 0.49% to 371.23%. Segmentation map accuracy between test and retest never dropped below 97%; the mean F1 scores were 0.85 for the intravisit dataset and 0.82 for the intervisit dataset. The best retest was achieved with disc-width regarding the Spearman correlation coefficient in both datasets. In terms of the Spearman correlation coefficient, the worst retests of the intervisit and intravisit groups were tortuosity density and artery tortuosity density, respectively. The intravisit group exhibited better retest reliability than the intervisit group (P < 0.001). Our linear model, with the two independent variables contrast-to-noise ratio and fractal dimension predicted the median retest reliability per image on its validation dataset, the intervisit group, with an R2 of 0.53 (P < 0.001).

Conclusions

Our findings highlight a considerable volatility in the reliability of some retinal biomarkers. Improving retest could allow disease progression modeling in smaller datasets or an individualized treatment approach. Image quality is moderately predictive of retest reliability, and further work is warranted to understand the reasons behind our observations better and thus ensure consistent retest results.

Subjects

MeSH terms (MEDLINE)

Retinal VesselsPhotographyHumansReproducibility of ResultsFemaleMaleMiddle AgedArtificial IntelligenceAgedFundus OculiImage Processing, Computer-Assisted

Authors and institutions

  • Department of Ophthalmology, Stadtspital Zürich, Zurich, Switzerland.
  • Spross Research Institute, Zurich, Switzerland.
  • Gutblick Research, Pfäffikon, Switzerland.
  • Department of Ophthalmology, Semmelweis University, Budapest, Hungary.
  • NIHR Biomedical Research Centre at Moorfields Eye Hospital NHS Foundation Trust, London, United Kingdom.
  • Institute of Ophthalmology, University College London, London, United Kingdom.

Funding

  • Medical Research Council

Cite

APA

Giesser, S. D., Turgut, F., Saad, A., Zoellin, J. R., Sommer, C., Zhou, Y., Wagner, S. K., Keane, P. A., Becker, M., DeBuc, D. C., & Somfai, G. M. (2024). Evaluating the Impact of Retinal Vessel Segmentation Metrics on Retest Reliability in a Clinical Setting: A Comparative Analysis Using AutoMorph. Investigative Ophthalmology & Visual Science, 65(13), 24. https://doi.org/10.1167/iovs.65.13.24

Vancouver

Giesser SD, Turgut F, Saad A, Zoellin JR, Sommer C, Zhou Y, et al. Evaluating the Impact of Retinal Vessel Segmentation Metrics on Retest Reliability in a Clinical Setting: A Comparative Analysis Using AutoMorph. Investigative Ophthalmology & Visual Science. 2024;65(13):24. doi:10.1167/iovs.65.13.24

BibTeX
@article{Giesser2024,
  author = {Giesser, Samuel D and Turgut, Ferhat and Saad, Amr and Zoellin, Jay R and Sommer, Chiara and Zhou, Yukun and Wagner, Siegfried K and Keane, Pearse A and Becker, Matthias and DeBuc, Delia Cabrera and Somfai, Gábor Márk},
  title = {Evaluating the Impact of Retinal Vessel Segmentation Metrics on Retest Reliability in a Clinical Setting: A Comparative Analysis Using AutoMorph},
  journal = {Investigative Ophthalmology & Visual Science},
  year = {2024},
  volume = {65},
  number = {13},
  pages = {24},
  doi = {10.1167/iovs.65.13.24}
}
Start for free →