EUVIMEDEUVIMED entdecken
Crossref

Are automated video interviews smart enough? Behavioral modes, reliability, validity, and bias of machine learning cognitive ability assessments.

Louis Hickman iD, Louis Tay, Sang Eun Woo

Journal of Applied Psychology · 2025-03 · 110 (3) · 314-335

DOI: 10.1037/apl0001236 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Applied Psychology, 2025-03. Verfasst von Louis Hickman und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, Virginia Tech. Hauptthemen laut MEDLINE: Machine Learning, Psychometrics, Video Recording, Aptitude. Die Arbeit wurde bisher 18-mal zitiert (Crossref). Gefördert von: National Science Foundation, Society for Industrial-Organizational Psychology Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automated video interviews (AVIs) that use machine learning (ML) algorithms to assess interviewees are increasingly popular. Extending prior AVI research focusing on noncognitive constructs, the present study critically evaluates the possibility of assessing cognitive ability with AVIs. By developing and examining AVI ML models trained to predict measures of three cognitive ability constructs (i.e., general mental ability, verbal ability, and intellect [as observed at zero acquaintance]), this research contributes to the literature in several ways. First, it advances our understanding of how cognitive abilities relate to interviewee behavior. Specifically, we found that verbal behaviors best predicted interviewee cognitive abilities, while neither paraverbal nor nonverbal behaviors provided incremental validity, suggesting that only verbal behaviors should be used to assess cognitive abilities. Second, across two samples of mock video interviews, we extensively evaluated the psychometric properties of the verbal behavior AVI ML model scores, including their reliability (internal consistency across interview questions and test-retest), validity (relationships with other variables and content), and fairness and bias (measurement and predictive). Overall, the general mental ability, verbal ability, and intellect AVI models captured similar behavioral manifestations of cognitive ability. Validity evidence results were mixed: For example, AVIs trained on observer-rated intellect exhibited superior convergent and criterion relationships (compared to the observer ratings they were trained to model) but had limited discriminant validity evidence. Our findings illustrate the importance of examining psychometric properties beyond convergence with the test that ML algorithms are trained to model. We provide recommendations for enhancing discriminant validity evidence in future AVIs. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Machine LearningPsychometricsVideo RecordingAptitudeVerbal BehaviorCognitionHumansAdultReproducibility of ResultsMaleFemaleYoung Adult

Autor:innen und Institutionen

  • Department of Psychology, Virginia Tech.
  • Department of Psychological Sciences, Purdue University.

ORCID-Profile: Louis Hickman

Förderung

  • National Science Foundation
  • Society for Industrial-Organizational Psychology Foundation

Zitieren

APA

Hickman, L., Tay, L., & Woo, S. E. (2025). Are automated video interviews smart enough? Behavioral modes, reliability, validity, and bias of machine learning cognitive ability assessments. Journal of Applied Psychology, 110(3), 314-335. https://doi.org/10.1037/apl0001236

Vancouver

Hickman L, Tay L, Woo SE. Are automated video interviews smart enough? Behavioral modes, reliability, validity, and bias of machine learning cognitive ability assessments. Journal of Applied Psychology. 2025;110(3):314-335. doi:10.1037/apl0001236

BibTeX
@article{Hickman2025,
  author = {Hickman, Louis and Tay, Louis and Woo, Sang Eun},
  title = {Are automated video interviews smart enough? Behavioral modes, reliability, validity, and bias of machine learning cognitive ability assessments},
  journal = {Journal of Applied Psychology},
  year = {2025},
  volume = {110},
  number = {3},
  pages = {314--335},
  doi = {10.1037/apl0001236}
}
Kostenlos starten →