EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Controlling Rater Effects in Divergent Thinking Assessment: An Item Response Theory Approach to Individual Response and Snapshot Scoring

Gerardo Pellegrino iD, Janika Saretzki iD, Mathias Benedek iD

Journal of Intelligence · 17.06.2025 · 13 (6) · 69

DOI: 10.3390/jintelligence13060069 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Intelligence, online erschienen am 17.06.2025. Verfasst von Gerardo Pellegrino und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of General Psychology, University of Padova, 35131 Padova, Italy. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: FWF Austrian Science Fund, Austrian Science Fund (FWF).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Scoring divergent thinking (DT) tasks poses significant challenges as differences between raters affect the resulting scores. Item Response Theory (IRT) offers a statistical framework to handle differences in rater severity and discrimination. We applied the IRT framework by re-analysing an open access dataset including three scored DT tasks from 202 participants. After comparing different IRT models, we examined rater severity and discrimination parameters for individual response scoring and snapshot scoring using the best-fitting model—Graded Response Model. Secondly, we compared IRT-adjusted scores with non-adjusted average and max-scoring scores in terms of reliability and fluency confound effect. Additionally, we simulated missing data to assess the robustness of these approaches. Our results showed that IRT models can be applied to both individual response scoring and snapshot scoring. IRT-adjusted and unadjusted scores were highly correlated, indicating that, under conditions of high inter-rater agreement, rater variability in severity and discrimination does not substantially impact scores. Overall, our study confirms that IRT is a valuable statistical framework for modeling rater severity and discrimination for different DT scores, although further research is needed to clarify the conditions under which it offers the greatest practical benefit.

Schlagwörter

Stichwörter der Autor:innen

consensual assessment techniquecreativity assessmentdivergent thinking assessmentitem response theoryjudge response theory

Autor:innen und Institutionen

  • Department of General Psychology, University of Padova, 35131 Padova, Italy.
  • Department of Psychology, University of Graz, 8010 Graz, Austria.
  • Department of Psychology, Charlotte Fresenius Hochschule München, 80797 Munich, Germany.
  • Department of Psychology, LMU Munich, 80802 Munich, Germany.

ORCID-Profile: Gerardo Pellegrino, Janika Saretzki, Mathias Benedek

Förderung

  • FWF Austrian Science Fund
  • Austrian Science Fund (FWF)

Zitieren

APA

Pellegrino, G., Saretzki, J., & Benedek, M. (2025). Controlling Rater Effects in Divergent Thinking Assessment: An Item Response Theory Approach to Individual Response and Snapshot Scoring. Journal of Intelligence, 13(6), 69. https://doi.org/10.3390/jintelligence13060069

Vancouver

Pellegrino G, Saretzki J, Benedek M. Controlling Rater Effects in Divergent Thinking Assessment: An Item Response Theory Approach to Individual Response and Snapshot Scoring. Journal of Intelligence. 2025;13(6):69. doi:10.3390/jintelligence13060069

BibTeX
@article{Pellegrino2025,
  author = {Pellegrino, Gerardo and Saretzki, Janika and Benedek, Mathias},
  title = {Controlling Rater Effects in Divergent Thinking Assessment: An Item Response Theory Approach to Individual Response and Snapshot Scoring},
  journal = {Journal of Intelligence},
  year = {2025},
  volume = {13},
  number = {6},
  pages = {69},
  doi = {10.3390/jintelligence13060069}
}
Kostenlos starten →