AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System
MIS Quarterly · 01.03.2026 · 50 (1) · 59-86
DOI: 10.25300/misq/2025/18946 ↗Kurz gefasst
Fachartikel in der Zeitschrift MIS Quarterly, online erschienen am 01.03.2026. Verfasst von Jean-Charles Pillet und 6 weiteren Autor:innen. Beteiligte Einrichtung: TBS Business School, 31000 Toulouse, France. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 114 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS.
Abstract (Originalsprache)
Content validation is an essential aspect of the scale development process that ensures that measurement instruments capture their intended constructs. However, researchers rarely undertake this core step in behavioral research because it requires costly data collection and specialized expertise. We present RATER (replicable approach to expert ratings), a free web-based system (www.contval.org) that can help the broader research community (scientists, reviewers, students) gain quick and reliable insights into the content validity of measurement instruments. Guided by psychometric measurement theory, RATER evaluates whether a scale’s items correspond to their intended construct, remain distinct from other constructs, and adequately represent all aspects of the construct’s content domain. The system employs two unique artificial intelligence models, RATERC and RATERD, which leverage psychometric scales from 2,443 journal articles spanning eight disciplines and two state-of-the-art large language model architectures (i.e., BERT and GPT). A set of six complementary studies confirms the RATER system’s accuracy, reliability, and usefulness. We find that RATER can augment the scale development and validation process, increasing the validity of findings in behavioral research.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- TBS Business School, 31000 Toulouse, France.
- Leeds School of Business, University of Colorado, Boulder, CO 80309 U.S.A.
- College of Business, Florida Atlantic University, Boca Raton, FL 33431 U.S.A.
- BI Norwegian Business School | Norwegian Defence University College, 0484 Oslo, Norway.
- Deakin Business School, Deakin University, Victoria 3125, Australia.
ORCID-Profile: Jean-Charles Pillet, Kai R Larsen, David Dobolyi, Magno Queiroz, Abram Handler, Jan Ketil Arnulf, Rajeev Sharma
Förderung
- NIA NIH HHS
Zitieren
APA
Pillet, J. C., Larsen, K. R., Dobolyi, D., Queiroz, M., Handler, A., Arnulf, J. K., & Sharma, R. (2026). AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System. MIS Quarterly, 50(1), 59-86. https://doi.org/10.25300/misq/2025/18946
Vancouver
Pillet JC, Larsen KR, Dobolyi D, Queiroz M, Handler A, Arnulf JK, et al. AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System. MIS Quarterly. 2026;50(1):59-86. doi:10.25300/misq/2025/18946
BibTeX
@article{Pillet2026,
author = {Pillet, Jean-Charles and Larsen, Kai R and Dobolyi, David and Queiroz, Magno and Handler, Abram and Arnulf, Jan Ketil and Sharma, Rajeev},
title = {AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System},
journal = {MIS Quarterly},
year = {2026},
volume = {50},
number = {1},
pages = {59--86},
doi = {10.25300/misq/2025/18946}
}
EUVIMED