Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A case study of Marie Skłodowska-Curie actions
PLOS ONE · 24.03.2025 · 20 (3) · e0317772
DOI: 10.1371/journal.pone.0317772 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS ONE, online erschienen am 24.03.2025. Verfasst von Rachel Heyard und 3 weiteren Autor:innen. Beteiligte Einrichtung: Center for Reproducible Science, Epidemiology, Biostatistic and Prevention Institute, University of Zurich, Zurich, Switzerland. Hauptthemen laut MEDLINE: Decision Making. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Croatian Science Foundation.
Abstract (Originalsprache)
Funding agencies rely on panel or consensus meetings to summarise individual evaluations of grant proposals into a final ranking. However, previous research has shown inconsistency in decisions and inefficiency of consensus meetings. Using data from the Marie Skłodowska-Curie Actions, we aimed at investigating the differences between an algorithmic approach to summarise the information from grant proposal individual evaluations to decisions after consensus meetings, and we present an exploratory comparative analysis. The algorithmic approach employed was a Bayesian hierarchical model resulting in a Bayesian ranking of the proposals using the individual evaluation reports cast prior to the consensus meeting. Parameters from the Bayesian hierarchical model and the subsequent ranking were compared to the scores, ranking and decisions established in the consensus meeting reports. The results from the evaluation of 1,006 proposals submitted to three panels (Life Science, Mathematics, Social Sciences and Humanities) in two call years (2015 and 2019) were investigated in detail. Overall, we found large discrepancies between the consensus reports and the scores a Bayesian hierarchical model would have predicted. The discrepancies were less pronounced when the scores were aggregated into funding rankings or decisions. The best agreement between the final funding ranking can be observed in the case of funding schemes with very low success rates. While we set out to understand if algorithmic approaches, with the aim of summarising individual evaluation scores, could replace consensus meetings, we concluded that currently individual scores assigned prior to the consensus meetings are not useful to predict the final funding outcomes of the proposals. Following our results, we would suggest to use individual evaluations for a triage and subsequently not discuss the weakest proposals in panel or consensus meetings. This would allow a more nuanced evaluation of a smaller set of proposals and help minimise the uncertainty and biases when allocating funding.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Center for Reproducible Science, Epidemiology, Biostatistic and Prevention Institute, University of Zurich, Zurich, Switzerland.
- European Research Executive Agency, European Commission, Brussels, Belgium.
- Department of Psychology, Faculty of Humanities and Social Sciences, University of Split, Split, Croatia.
- Center for Evidence-based Medicine, Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia.
ORCID-Profile: Rachel Heyard, David G Pina, Ana Marušić
Förderung
- Croatian Science Foundation
Zitieren
APA
Heyard, R., Pina, D. G., Buljan, I., & Marušić, A. (2025). Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A case study of Marie Skłodowska-Curie actions. PLOS ONE, 20(3), e0317772. https://doi.org/10.1371/journal.pone.0317772
Vancouver
Heyard R, Pina DG, Buljan I, Marušić A. Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A case study of Marie Skłodowska-Curie actions. PLOS ONE. 2025;20(3):e0317772. doi:10.1371/journal.pone.0317772
BibTeX
@article{Heyard2025,
author = {Heyard, Rachel and Pina, David G and Buljan, Ivan and Marušić, Ana},
title = {Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A case study of Marie Skłodowska-Curie actions},
journal = {PLOS ONE},
year = {2025},
volume = {20},
number = {3},
pages = {e0317772},
doi = {10.1371/journal.pone.0317772}
}
EUVIMED