EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Automating the quality prediction of survey questions: predicting measurement quality directly from natural text

Tiancheng Yang, Matthias Schonlau, Lydia Repke, Barbara Felderer, Ilia Sucholutsky

Journal of the Royal Statistical Society Series A: Statistics in Society · 28.04.2026

DOI: 10.1093/jrsssa/qnag058 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the Royal Statistical Society Series A: Statistics in Society, online erschienen am 28.04.2026. Verfasst von Tiancheng Yang und 4 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: SSHRC.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The survey quality predictor (SQP) is a web-based tool designed to predict the measurement quality of survey questions based on up to 72 manually coded formal and linguistic characteristics (e.g. domain, response scale properties, linguistic complexity). Users must input these features following a detailed coding manual, after which a trained random forest (RF) model predicts measurement quality. Here, we evaluate whether measurement quality can instead be predicted directly from the natural language text of a survey question, eliminating the need for manual coding. We find a fine-tuned language model can predict survey item quality based solely on the question and answer options, achieving performance comparable to the RF model currently implemented in SQP, which is trained on manually coded features. Specifically, we fine-tuned xlm-RoBERTa, a multilingual transformer-based model trained on multiple text corpora in over 100 languages, using the SQP dataset. Our findings suggest the current SQP web interface (https://sqp.gesis.org), which requires users to manually code the 72 features, can be simplified. A redesigned, more user-friendly web interface could allow users to only enter the survey question and answer options, with the model automatically predicting measurement quality. This would enhance accessibility and lower the barrier for researchers and practitioners using SQP.

Förderung

  • SSHRC

Zitieren

APA

Yang, T., Schonlau, M., Repke, L., Felderer, B., & Sucholutsky, I. (2026). Automating the quality prediction of survey questions: predicting measurement quality directly from natural text. Journal of the Royal Statistical Society Series A: Statistics in Society. https://doi.org/10.1093/jrsssa/qnag058

Vancouver

Yang T, Schonlau M, Repke L, Felderer B, Sucholutsky I. Automating the quality prediction of survey questions: predicting measurement quality directly from natural text. Journal of the Royal Statistical Society Series A: Statistics in Society. 2026. doi:10.1093/jrsssa/qnag058

BibTeX
@article{Yang2026,
  author = {Yang, Tiancheng and Schonlau, Matthias and Repke, Lydia and Felderer, Barbara and Sucholutsky, Ilia},
  title = {Automating the quality prediction of survey questions: predicting measurement quality directly from natural text},
  journal = {Journal of the Royal Statistical Society Series A: Statistics in Society},
  year = {2026},
  doi = {10.1093/jrsssa/qnag058}
}
Kostenlos starten →