How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores.
Journal of Applied Psychology · 2023-08 · 108 (8) · 1277-1299
DOI: 10.1037/apl0001082 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Applied Psychology, 2023-08. Verfasst von Jinyan Fan und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychological Sciences, Auburn University. Hauptthemen laut MEDLINE: Artificial Intelligence, Personality Disorders. Die Arbeit wurde bisher 93-mal zitiert (Crossref).
Abstract (Originalsprache)
The present study explores the plausibility of measuring personality indirectly through an artificial intelligence (AI) chatbot. This chatbot mines various textual features from users' free text responses collected during an online conversation/interview and then uses machine learning algorithms to infer personality scores. We comprehensively examine the psychometric properties of the machine-inferred personality scores, including reliability (internal consistency, split-half, and test-retest), factorial validity, convergent and discriminant validity, and criterion-related validity. Participants were undergraduate students (n = 1,444) enrolled in a large southeastern public university in the United States who completed a self-report Big Five personality measure (IPIP-300) and engaged with an AI chatbot for approximately 20-30 min. In a subsample (n = 407), we obtained participants' cumulative grade point averages from the University Registrar and had their peers rate their college adjustment. In an additional sample (n = 61), we obtained test-retest data. Results indicated that machine-inferred personality scores (a) had overall acceptable reliability at both the domain and facet levels, (b) yielded a comparable factor structure to self-reported questionnaire-derived personality scores, (c) displayed good convergent validity but relatively poor discriminant validity (averaged convergent correlations = .48 vs. averaged machine-score correlations = .35 in the test sample), (d) showed low criterion-related validity, and (e) exhibited incremental validity over self-reported questionnaire-derived personality scores in some analyses. In addition, there was strong evidence for cross-sample generalizability of psychometric properties of machine scores. Theoretical implications, future research directions, and practical considerations are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Psychological Sciences, Auburn University.
- Department of Psychological Sciences, Kansas State University.
- School of Labor and Employment Relations, University of Illinois Urbana-Champaign.
- School of Information Systems and Management, Muma College of Business, University of South Florida-St. Petersburg.
- Department of Behavioral Sciences and Leadership, United States Air Force Academy.
ORCID-Profile: Jinyan Fan, Tianjun Sun, Jiayi Liu, Teng Zhao, Bo Zhang, Zheng Chen, Elissa Hack
Zitieren
APA
Fan, J., Sun, T., Liu, J., Zhao, T., Zhang, B., Chen, Z., Glorioso, M., & Hack, E. (2023). How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores. Journal of Applied Psychology, 108(8), 1277-1299. https://doi.org/10.1037/apl0001082
Vancouver
Fan J, Sun T, Liu J, Zhao T, Zhang B, Chen Z, et al. How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores. Journal of Applied Psychology. 2023;108(8):1277-1299. doi:10.1037/apl0001082
BibTeX
@article{Fan2023,
author = {Fan, Jinyan and Sun, Tianjun and Liu, Jiayi and Zhao, Teng and Zhang, Bo and Chen, Zheng and Glorioso, Melissa and Hack, Elissa},
title = {How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores},
journal = {Journal of Applied Psychology},
year = {2023},
volume = {108},
number = {8},
pages = {1277--1299},
doi = {10.1037/apl0001082}
}
EUVIMED