EUVIMEDEUVIMED entdecken
Open Access

AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System

Jean-Charles Pillet iD, Kai R Larsen iD, David Dobolyi iD, Magno Queiroz iD, Abram Handler iD, Jan Ketil Arnulf iD, Rajeev Sharma iD

MIS Quarterly · 01.03.2026 · 50 (1) · 59-86

DOI: 10.25300/misq/2025/18946 ↗

Kurz gefasst

Fachartikel in der Zeitschrift MIS Quarterly, online erschienen am 01.03.2026. Verfasst von Jean-Charles Pillet und 6 weiteren Autor:innen. Beteiligte Einrichtung: TBS Business School, 31000 Toulouse, France. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 114 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Content validation is an essential aspect of the scale development process that ensures that measurement instruments capture their intended constructs. However, researchers rarely undertake this core step in behavioral research because it requires costly data collection and specialized expertise. We present RATER (replicable approach to expert ratings), a free web-based system (www.contval.org) that can help the broader research community (scientists, reviewers, students) gain quick and reliable insights into the content validity of measurement instruments. Guided by psychometric measurement theory, RATER evaluates whether a scale’s items correspond to their intended construct, remain distinct from other constructs, and adequately represent all aspects of the construct’s content domain. The system employs two unique artificial intelligence models, RATERC and RATERD, which leverage psychometric scales from 2,443 journal articles spanning eight disciplines and two state-of-the-art large language model architectures (i.e., BERT and GPT). A set of six complementary studies confirms the RATER system’s accuracy, reliability, and usefulness. We find that RATER can augment the scale development and validation process, increasing the validity of findings in behavioral research.

Schlagwörter

Stichwörter der Autor:innen

behavioral researchcontent validitydesign science researchlarge language models (LLMs)machine learningpsychometricsresearch rigorscale development

Autor:innen und Institutionen

  • TBS Business School, 31000 Toulouse, France.
  • Leeds School of Business, University of Colorado, Boulder, CO 80309 U.S.A.
  • College of Business, Florida Atlantic University, Boca Raton, FL 33431 U.S.A.
  • BI Norwegian Business School | Norwegian Defence University College, 0484 Oslo, Norway.
  • Deakin Business School, Deakin University, Victoria 3125, Australia.

ORCID-Profile: Jean-Charles Pillet, Kai R Larsen, David Dobolyi, Magno Queiroz, Abram Handler, Jan Ketil Arnulf, Rajeev Sharma

Förderung

  • NIA NIH HHS

Zitieren

APA

Pillet, J. C., Larsen, K. R., Dobolyi, D., Queiroz, M., Handler, A., Arnulf, J. K., & Sharma, R. (2026). AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System. MIS Quarterly, 50(1), 59-86. https://doi.org/10.25300/misq/2025/18946

Vancouver

Pillet JC, Larsen KR, Dobolyi D, Queiroz M, Handler A, Arnulf JK, et al. AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System. MIS Quarterly. 2026;50(1):59-86. doi:10.25300/misq/2025/18946

BibTeX
@article{Pillet2026,
  author = {Pillet, Jean-Charles and Larsen, Kai R and Dobolyi, David and Queiroz, Magno and Handler, Abram and Arnulf, Jan Ketil and Sharma, Rajeev},
  title = {AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System},
  journal = {MIS Quarterly},
  year = {2026},
  volume = {50},
  number = {1},
  pages = {59--86},
  doi = {10.25300/misq/2025/18946}
}
Kostenlos starten →