EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process

Hannah Fraser iD, Martin Bush iD, Bonnie C Wintle, Fallon Mody, Eden T Smith, Anca M Hanea iD, Elliot Gould iD, Victoria Hemming, Daniel G Hamilton, Libby Rumpff, David P Wilkinson iD, Ross Pearson und 19 weitere

PLOS ONE · 2023

DOI: 10.1371/journal.pone.0274429 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, 2023. Verfasst von Hannah Fraser und 30 weiteren Autor:innen. Beteiligte Einrichtung: MetaMelb Lab, University of Melbourne, Melbourne, Victoria, Australia. Hauptthemen laut MEDLINE: Data Accuracy, Behavioral Sciences. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Advanced Research Projects Agency.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Data AccuracyBehavioral SciencesHumansReproducibility of ResultsCosts and Cost AnalysisPeer Review

Autor:innen und Institutionen

  • MetaMelb Lab, University of Melbourne, Melbourne, Victoria, Australia.
  • Quantitative & Applied Ecology Group, University of Melbourne, Melbourne, Victoria, Australia.
  • Centre of Excellence for Biosecurity Risk Analysis, University of Melbourne, Melbourne, Victoria, Australia.
  • Martin Conservation Decisions Lab, Department of Forest and Conservation Sciences, University of British Columbia, Vancouver, Canada.
  • School of Natural and Environmental Sciences, Newcastle University, Newcastle upon Tyne, United Kingdom.
  • Centre for Argument Technology, School of Science and Engineering, University of Dundee, Dundee, United Kingdom.

ORCID-Profile: Hannah Fraser, Martin Bush, Anca M Hanea, Elliot Gould, David P Wilkinson, Aaron Willcox, Alexandru Marcoci

Förderung

  • Defense Advanced Research Projects Agency

Zitieren

APA

Fraser, H., Bush, M., Wintle, B. C., Mody, F., Smith, E. T., Hanea, A. M., Gould, E., Hemming, V., Hamilton, D. G., Rumpff, L., Wilkinson, D. P., Pearson, R., Singleton Thorn, F., Ashton, R., Willcox, A., Gray, C. T., Head, A., Ross, M., Groenewegen, R., …, & Fidler, F. (2023). Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. PLOS ONE. https://doi.org/10.1371/journal.pone.0274429

Vancouver

Fraser H, Bush M, Wintle BC, Mody F, Smith ET, Hanea AM, et al. Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process. PLOS ONE. 2023. doi:10.1371/journal.pone.0274429

BibTeX
@article{Fraser2023,
  author = {Fraser, Hannah and Bush, Martin and Wintle, Bonnie C and Mody, Fallon and Smith, Eden T and Hanea, Anca M and Gould, Elliot and Hemming, Victoria and Hamilton, Daniel G and Rumpff, Libby and Wilkinson, David P and Pearson, Ross and Singleton Thorn, Felix and Ashton, Raquel and Willcox, Aaron and Gray, Charles T and Head, Andrew and Ross, Melissa and Groenewegen, Rebecca and Marcoci, Alexandru and Vercammen, Ans and Parker, Timothy H and Hoekstra, Rink and Nakagawa, Shinichi and Mandel, David R and van Ravenzwaaij, Don and McBride, Marissa and Sinnott, Richard O and Vesk, Peter and Burgman, Mark and Fidler, Fiona},
  title = {Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process},
  journal = {PLOS ONE},
  year = {2023},
  doi = {10.1371/journal.pone.0274429}
}
Kostenlos starten →