EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Scoring German Alternate Uses Items Applying Large Language Models

Janika Saretzki iD, Thomas Knopf iD, Boris Forthmann iD, Benjamin Goecke iD, Ann-Kathrin Jaggy iD, Mathias Benedek iD, Selina Weiss iD

Journal of Intelligence · 29.05.2025 · 13 (6) · 64

DOI: 10.3390/jintelligence13060064 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Intelligence, online erschienen am 29.05.2025. Verfasst von Janika Saretzki und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, University of Graz, 8010 Graz, Austria. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: FWF Austrian Science Fund, Austrian Science Fund (FWF).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The alternate uses task (AUT) is the most popular measure when it comes to the assessment of creative potential. Since their implementation, AUT responses have been rated by humans, which is a laborious task and requires considerable resources. Large language models (LLMs) have shown promising performance in automatically scoring AUT responses in English as well as in other languages, but it is not clear which method works best for German data. Therefore, we investigated the performance of different LLMs for the automated scoring of German AUT responses. We compiled German data across five research groups including ~50,000 responses for 15 different alternate uses objects from eight lab and online survey studies (including ~2300 participants) to examine generalizability across datasets and assessment conditions. Following a pre-registered analysis plan, we compared the performance of two fine-tuned, multilingual LLM-based approaches [Cross-Lingual Alternate Uses Scoring (CLAUS) and the Open Creativity Scoring with Artificial Intelligence (OCSAI)] with the Generative Pre-trained Transformer (GPT-4) in scoring (a) the original German AUT responses and (b) the responses translated to English. We found that the LLM-based scorings were substantially correlated with human ratings, with higher relationships for OCSAI followed by GPT-4 and CLAUS. Response translation, however, had no consistent positive effect. We discuss the generalizability of the results across different items and studies and derive recommendations and future directions.

Schlagwörter

Stichwörter der Autor:innen

GPTGermanalternate uses taskassessmentautomated scoringcreativitydivergent thinkinglarge language models

Autor:innen und Institutionen

  • Department of Psychology, University of Graz, 8010 Graz, Austria.
  • Department of Psychology, Charlotte Fresenius Hochschule München, 80797 Munich, Germany.
  • Department of Psychology, LMU Munich, 80802 Munich, Germany.
  • GESIS-Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany.
  • Institute of Psychology, University of Münster, 48149 Münster, Germany.
  • Hector Research Institute of Education Sciences and Psychology, University of Tübingen, 72072 Tübingen, Germany.

ORCID-Profile: Janika Saretzki, Thomas Knopf, Boris Forthmann, Benjamin Goecke, Ann-Kathrin Jaggy, Mathias Benedek, Selina Weiss

Förderung

  • FWF Austrian Science Fund
  • Austrian Science Fund (FWF)

Zitieren

APA

Saretzki, J., Knopf, T., Forthmann, B., Goecke, B., Jaggy, A. K., Benedek, M., & Weiss, S. (2025). Scoring German Alternate Uses Items Applying Large Language Models. Journal of Intelligence, 13(6), 64. https://doi.org/10.3390/jintelligence13060064

Vancouver

Saretzki J, Knopf T, Forthmann B, Goecke B, Jaggy AK, Benedek M, et al. Scoring German Alternate Uses Items Applying Large Language Models. Journal of Intelligence. 2025;13(6):64. doi:10.3390/jintelligence13060064

BibTeX
@article{Saretzki2025,
  author = {Saretzki, Janika and Knopf, Thomas and Forthmann, Boris and Goecke, Benjamin and Jaggy, Ann-Kathrin and Benedek, Mathias and Weiss, Selina},
  title = {Scoring German Alternate Uses Items Applying Large Language Models},
  journal = {Journal of Intelligence},
  year = {2025},
  volume = {13},
  number = {6},
  pages = {64},
  doi = {10.3390/jintelligence13060064}
}
Kostenlos starten →