EUVIMEDEUVIMED entdecken
Open Access

Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction

Julina Maharjan, Ruoming Jin, Jianfeng Zhu, Deric Kenne

Journal of Medical Internet Research · 08.07.2025 · 27 · e75347-e75347

DOI: 10.2196/75347 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 08.07.2025. Verfasst von Julina Maharjan und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Kent State University, 800 East Summit Street, Kent, OH, 44242, United States, 1 3305931365. Hauptthemen laut MEDLINE: Psychometrics, Personality, Language. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Recent advancements in large language models (LLMs) have generated significant interest in their potential for assessing psychological constructs, particularly personality traits. While prior research has explored LLMs’ capabilities in zero-shot or few-shot personality inference, few studies have systematically evaluated LLM embeddings within a psychometric validity framework or examined their correlations with linguistic and emotional markers. Additionally, the comparative efficacy of LLM embeddings against traditional feature engineering methods remains underexplored, leaving gaps in understanding their scalability and interpretability for computational personality assessment.

Objective

This study evaluates LLM embeddings for personality trait prediction through four key analyses: (1) performance comparison with zero-shot methods on PANDORA Reddit data, (2) psychometric validation and correlation with LIWC (Linguistic Inquiry and Word Count) and emotion features, (3) benchmarking against traditional feature engineering approaches, and (4) assessment of model size effects (OpenAI vs BERT vs RoBERTa). We aim to establish LLM embeddings as a psychometrically valid and efficient alternative for personality assessment.

Methods

We conducted a multistage analysis using 1 million Reddit posts from the PANDORA Big Five personality dataset. First, we generated text embeddings using 3 LLM architectures (RoBERTa, BERT, and OpenAI) and trained a custom bidirectional long short-term memory model for personality prediction. We compared this approach against zero-shot inference using prompt-based methods. Second, we extracted psycholinguistic features (LIWC categories and National Research Council emotions) and performed feature engineering to evaluate potential performance enhancements. Third, we assessed the psychometric validity of LLM embeddings: reliability validity using Cronbach α and convergent validity analysis by examining correlations between embeddings and established linguistic markers. Finally, we performed traditional feature engineering on static psycholinguistic features to assess performance under different settings.

Results

LLM embeddings trained using simple deep learning techniques significantly outperform zero-shot approaches on average by 45% across all personality traits. Although psychometric validation tests indicate moderate reliability, with an average Cronbach α of 0.63, correlation analyses spark a strong association with key linguistic or emotional markers; openness correlates highly with social ( r =0.53), conscientiousness with linguistic ( r =0.46), extraversion with social ( r =0.41), agreeableness with pronoun usage ( r =0.40), and neuroticism with politics-related text ( r =0.63). Despite adding advanced feature engineering on linguistic features, the performance did not improve, suggesting that LLM embeddings inherently capture key linguistic features. Furthermore, our analyses demonstrated efficacy on larger model size with a computational cost trade-off.

Conclusions

Our findings demonstrate that LLM embeddings offer a robust alternative to zero-shot methods in personality trait analysis, capturing key linguistic patterns without requiring extensive feature engineering. The correlation between established psycholinguistic markers and the performance trade-off with computational cost provides a hint for future computational linguistic work targeting LLM for personality assessment. Further research should explore fine-tuning strategies to enhance psychometric validity.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PsychometricsPersonalityLanguageHumansEmotionsLarge Language Models

Stichwörter der Autor:innen

Big Five personalityRedditartificial intelligencedeep learningembeddingslarge language modelspsychologysocial media

Autor:innen und Institutionen

  • Department of Computer Science, Kent State University, 800 East Summit Street, Kent, OH, 44242, United States, 1 3305931365.
  • Department of Public Health, Kent State University, Kent, OH, United States.

Zitieren

APA

Maharjan, J., Jin, R., Zhu, J., & Kenne, D. (2025). Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction. Journal of Medical Internet Research, 27, e75347-e75347. https://doi.org/10.2196/75347

Vancouver

Maharjan J, Jin R, Zhu J, Kenne D. Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction. Journal of Medical Internet Research. 2025;27:e75347-e75347. doi:10.2196/75347

BibTeX
@article{Maharjan2025,
  author = {Maharjan, Julina and Jin, Ruoming and Zhu, Jianfeng and Kenne, Deric},
  title = {Psychometric Evaluation of Large Language Model Embeddings for Personality Trait Prediction},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e75347--e75347},
  doi = {10.2196/75347}
}
Kostenlos starten →