Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models
Behavioral Sciences · 04.08.2026 · 16 (8) · 1338
DOI: 10.3390/bs16081338 ↗Kurz gefasst
Fachartikel in der Zeitschrift Behavioral Sciences, online erschienen am 04.08.2026. Verfasst von Nicola Milano und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The present study examines whether large language model (LLM) embeddings can approximate the psychometric validation process of a newly developed psychological scale. Using the Academic Psychological Distress Scale (APDS) as a case study, we replicated the full validation pipeline, traditionally performed on human response data, by applying exploratory and confirmatory factor analyses (EFA and CFA) to semantic similarity matrices derived from LLM embeddings. We first computed cosine similarity among all 58 APDS items and found a moderately strong correspondence with the participants’ correlation matrix (ρ = 0.57). An EFA performed on the embedding-based similarity matrix yielded a six-factor solution explaining 79.1% of the variance. The best-performing model (32 items, loading cutoff = 0.70) demonstrated acceptable model fit when tested on participants’ responses (CFI = 0.88; TLI = 0.87; RMSEA = 0.07–0.08; SRMR = 0.06) numerically comparable to the fit obtained in the original human-based validation based on 367 participants’ response. Reliability estimates were satisfactory for most factors (ω/α = 0.82–0.95), although one factor showed lower internal consistency. The percentage of overlap between human-derived and embedding-derived structures, indicated an overall structural correspondence of 64%, with factor-level overlap ranging from 50% to 100%. While embeddings reproduced most dimensions of academic distress, they failed to recover the demoralization factor and instead introduced an additional behavioral cluster. These findings suggest that LLM embeddings can capture substantial aspects of the latent structure of psychological constructs and can support early-stage, low-data, embedding-first scale development. We propose a linguistic validity pipeline as a methodological framework for integrating semantic embeddings into psychometric validation procedures, highlighting their promise as a complementary tool for human-centered assessment.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Humanistic Studies, University of Naples Federico II, Via Porta di Massa 1, 80125 Naples, Italy.
ORCID-Profile: Nicola Milano, Davide Marocco
Zitieren
APA
Milano, N., Pizzo, R., Scandurra, C., Freda, M. F., & Marocco, D. (2026). Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models. Behavioral Sciences, 16(8), 1338. https://doi.org/10.3390/bs16081338
Vancouver
Milano N, Pizzo R, Scandurra C, Freda MF, Marocco D. Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models. Behavioral Sciences. 2026;16(8):1338. doi:10.3390/bs16081338
BibTeX
@article{Milano2026,
author = {Milano, Nicola and Pizzo, Rosa and Scandurra, Cristiano and Freda, Maria Francesca and Marocco, Davide},
title = {Deep Semantics Analysis for Scale Development: Replicating Psychometrics Validation Process with Large Language Models},
journal = {Behavioral Sciences},
year = {2026},
volume = {16},
number = {8},
pages = {1338},
doi = {10.3390/bs16081338}
}
EUVIMED