EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Evaluating the predictive performance of presence–absence models: Why can the same model appear excellent or poor?

Nerea Abrego iD, Otso Ovaskainen iD

Ecology and Evolution · 2023-12 · 13 (12)

DOI: 10.1002/ece3.10784 ↗

At a glance

Journal article in Ecology and Evolution, published online on 2023-12-18. Written by Nerea Abrego and 1 co-authors. Institution: Department of Biological and Environmental Science University of Jyväskylä Jyväskylä Finland. Cited 35 times so far (Crossref); 45 references. The full text is freely available (open access, CC BY 4.0). Funded by: Academy of Finland, H2020 European Research Council, Jane ja Aatos Erkon Säätiö.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

When comparing multiple models of species distribution, models yielding higher predictive performance are clearly to be favored. A more difficult question is how to decide whether even the best model is “good enough”. Here, we clarify key choices and metrics related to evaluating the predictive performance of presence–absence models. We use a hierarchical case study to evaluate how four metrics of predictive performance (AUC, Tjur's R 2 , max‐Kappa, and max‐TSS) relate to each other, the random and fixed effects parts of the model, the spatial scale at which predictive performance is measured, and the cross‐validation strategy chosen. We demonstrate that the very same metric can achieve different values for the very same model, even when similar cross‐validation strategies are followed, depending on the spatial scale at which predictive performance is measured. Among metrics, Tjur's R 2 and max‐Kappa generally increase with species' prevalence, whereas AUC and max‐TSS are largely independent of prevalence. Thus, Tjur's R 2 and max‐Kappa often reach lower values when measured at the smallest scales considered in the study, while AUC and max‐TSS reaching similar values across the different spatial levels included in the study. However, they provide complementary insights on predictive performance. The very same model may appear excellent or poor not only due to the applied metric, but also how predictive performance is exactly calculated, calling for great caution on the interpretation of predictive performance. The most comprehensive evaluation of predictive performance can be obtained by evaluating predictive performance through the combination of measures providing complementary insights. Instead of following simple rules of thumb or focusing on absolute values, we recommend comparing the achieved predictive performance to the researcher's own a priori expectations on how easy it is to make predictions related to the same question that the model is used for.

Subjects

Author keywords

AUCTjur's R2accuracycross‐validationdiscriminationjoint species distribution modelmax‐TSSmax‐kappapresence–absence modelsensitivityspecificity

Authors and institutions

  • Department of Biological and Environmental Science University of Jyväskylä Jyväskylä Finland.
  • Department of Agricultural Sciences University of Helsinki Helsinki Finland.
  • Department of Biology, Centre for Biodiversity Dynamics Norwegian University of Science and Technology Trondheim Norway.
  • Organismal and Evolutionary Biology Research Programme, Faculty of Biological and Environmental Sciences University of Helsinki Helsinki Finland.

ORCID profiles: Nerea Abrego, Otso Ovaskainen

Funding

  • Academy of Finland
  • H2020 European Research Council
  • Jane ja Aatos Erkon Säätiö
  • Norges Forskningsråd

Cite

APA

Abrego, N., & Ovaskainen, O. (2023). Evaluating the predictive performance of presence–absence models: Why can the same model appear excellent or poor?. Ecology and Evolution, 13(12). https://doi.org/10.1002/ece3.10784

Vancouver

Abrego N, Ovaskainen O. Evaluating the predictive performance of presence–absence models: Why can the same model appear excellent or poor?. Ecology and Evolution. 2023;13(12). doi:10.1002/ece3.10784

BibTeX
@article{Abrego2023,
  author = {Abrego, Nerea and Ovaskainen, Otso},
  title = {Evaluating the predictive performance of presence–absence models: Why can the same model appear excellent or poor?},
  journal = {Ecology and Evolution},
  year = {2023},
  volume = {13},
  number = {12},
  doi = {10.1002/ece3.10784}
}
Start for free →