AIPsychoBench: Understanding the Psychometric Differences Between LLMs and Humans
Topics in Cognitive Science · 2026-04 · 18 (2)
DOI: 10.1111/tops.70041 ↗Kurz gefasst
Fachartikel in der Zeitschrift Topics in Cognitive Science, online erschienen am 09.03.2026. Verfasst von Wei Xie und 7 weiteren Autor:innen. Beteiligte Einrichtung: College of Computer Science and Technology, National University of Defense Technology. Hauptthemen laut MEDLINE: Large Language Models, Psychometrics. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large Language Models (LLMs) with hundreds of billions of parameters have exhibited human‐like intelligence by learning from vast amounts of internet‐scale data. However, the uninterpretability of large‐scale neural networks raises concerns about the reliability of LLM. Studies have attempted to assess the psychometric properties of LLMs by borrowing concepts from human psychology to enhance their interpretability, but they fail to account for the fundamental differences between LLMs and humans. This results in high rejection rates when human scales are reused directly. Furthermore, these scales do not support the measurement of LLM psychological property variations in different languages. This paper introduces AIPsychoBench, a specialized benchmark tailored to assess the psychological properties of LLM. It uses a lightweight role‐playing prompt to bypass LLM alignment, improving the average effective response rate from 70.12% to 90.40%. Meanwhile, the average biases are only 3.3% (positive) and 2.1% (negative), which are significantly lower than the biases of 9.8% and 6.9%, respectively, caused by traditional jailbreak prompts. Furthermore, among the total of 112 psychometric subcategories, the score deviations for seven languages compared to English ranged from 5% to 20.2% in 43 subcategories, providing the first comprehensive evidence of the linguistic impact on the psychometrics of LLM.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Computer Science and Technology, National University of Defense Technology.
- Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR).
- Institute of Information Engineering, Chinese Academy of Sciences.
Zitieren
APA
Xie, W., Wang, Z., Ma, S., Sun, X., Chen, K., Wang, E., Liu, W., & Tong, H. (2026). AIPsychoBench: Understanding the Psychometric Differences Between LLMs and Humans. Topics in Cognitive Science, 18(2). https://doi.org/10.1111/tops.70041
Vancouver
Xie W, Wang Z, Ma S, Sun X, Chen K, Wang E, et al. AIPsychoBench: Understanding the Psychometric Differences Between LLMs and Humans. Topics in Cognitive Science. 2026;18(2). doi:10.1111/tops.70041
BibTeX
@article{Xie2026,
author = {Xie, Wei and Wang, Zhenhua and Ma, Shuoyoucheng and Sun, Xiaobing and Chen, Kai and Wang, Enze and Liu, Wei and Tong, Hanying},
title = {AIPsychoBench: Understanding the Psychometric Differences Between LLMs and Humans},
journal = {Topics in Cognitive Science},
year = {2026},
volume = {18},
number = {2},
doi = {10.1111/tops.70041}
}
EUVIMED