Do peers share the same criteria for assessing grant applications?
Research Evaluation · 03.01.2022 · 31 (1) · 104-117
DOI: 10.1093/reseval/rvab034 ↗Kurz gefasst
Fachartikel in der Zeitschrift Research Evaluation, online erschienen am 02.12.2021. Verfasst von Sven E Hug und 1 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: swissuniversities, SUK Program P-3 ‘Performances de la recherche en sciences humaines et sociales, University of Zurich.
Abstract (Originalsprache)
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in Switzerland on 23 grant review criteria. We have employed latent class tree modelling to identify subgroups in which scholars rated criteria similarly (i.e. latent classes) and to explore covariates predicting class membership. We have identified two consensus classes, two consensus-close classes, and a consensus-far class. The consensus classes contain a core consensus (10 criteria related to knowledge gaps; feasibility; rigour; comprehensibility and argumentation; academic relevance; competence and experience of the applicant) and a broad consensus that include the core consensus plus eight contribution-related criteria, such as originality. These results provide a possible explanation for the disagreement effect. Moreover, the results are consistent with the notion of conservatism, which holds that original research is undervalued in peer review, while other aspects, such as methodology and feasibility, are overweighted. The covariate analysis indicated that age and having tenure increase from the consensus far to the consensus close to the consensus classes. This suggests that the more academic experience scholars accumulate, the more their understanding of review criteria conforms to the social norm.
Förderung
- swissuniversities, SUK Program P-3 ‘Performances de la recherche en sciences humaines et sociales
- University of Zurich
Zitieren
APA
Hug, S. E., & Ochsner, M. (2022). Do peers share the same criteria for assessing grant applications?. Research Evaluation, 31(1), 104-117. https://doi.org/10.1093/reseval/rvab034
Vancouver
Hug SE, Ochsner M. Do peers share the same criteria for assessing grant applications?. Research Evaluation. 2022;31(1):104-117. doi:10.1093/reseval/rvab034
BibTeX
@article{Hug2022,
author = {Hug, Sven E and Ochsner, Michael},
title = {Do peers share the same criteria for assessing grant applications?},
journal = {Research Evaluation},
year = {2022},
volume = {31},
number = {1},
pages = {104--117},
doi = {10.1093/reseval/rvab034}
}
EUVIMED