Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications
JMIR Human Factors · 2026-10-21 · 13 · e102910-e102910
DOI: 10.2196/102910 ↗At a glance
Validierungsstudie in JMIR Human Factors, published online on 2026-10-21. Written by Leming Zhou and 7 co-authors. Institution: Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, 3396 Fifth Avenue, Pittsburgh, P…. Major MEDLINE topics: Large Language Models, Psychometrics, Privacy. 44 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Privacy concerns represent a primary barrier to the adoption of large language model (LLM) applications, particularly in sensitive domains such as health care. Although a growing body of survey-based research has examined user privacy concerns with LLMs, no standardized instrument exists for measuring this construct.
Objective
This study aimed to develop and validate a concise, standardized questionnaire for assessing user privacy concerns with LLM-based systems, suitable for use by researchers and practitioners such as AI application developers and health care professionals.
Methods
A literature review was conducted to identify privacy-related domains relevant to LLM use. A draft questionnaire was developed and subjected to content validity review by 8 experts. The finalized instrument was administered via Qualtrics to adults aged 18 years or older who were aware of LLM-based applications, recruited through the University of Pittsburgh, LinkedIn, and the Pitt+Me research participant registry between April 2025 and July 2026. Internal consistency reliability and factor structure were assessed, with factor structure examined in a split-sample design, in which exploratory factor analysis (EFA) was conducted on a random one-third of the sample and confirmatory factor analysis (CFA) on the holdout two-thirds.
Results
All questionnaire items achieved acceptable content validity (item-level content validity index >0.80; average scale-level content validity index=0.964). A total of 603 responses were collected. After preprocessing, the final sample comprised 361 participants. Most participants were younger than 35 years of age (198/361, 54.8%), female (230/361, 63.7%), and White (285/361, 78.9%); 88.1% (318/361) held at least a bachelor’s degree. The 6-item questionnaire showed high internal consistency (Cronbach α=0.90; McDonald ω=0.90). EFA (n=121) supported a single dominant factor with all standardized loadings ≥0.70. CFA (n=240) showed excellent fit (comparative fit index=0.996; Tucker-Lewis index=0.993; root mean square error of approximation=0.046; standardized root mean square residual=0.057). A 2-factor model separating the transparency- and control-related items fit significantly better than the 1-factor model (Δ χ 2 1 =8.9; P =.003), but the 2 factors were correlated at r =0.83, indicating closely related facets. Because the Fornell-Larcker criterion was met for the transparency and control subscale but not for data use and risk salience, the 6-item total score remains the recommended primary metric.
Conclusions
The LLM privacy concern assessment is a concise, psychometrically sound 6-item instrument that measures privacy concern with LLM-based systems via 2 correlated subdimensions (transparency and control; data use and risk salience) summarized by a single total score. It enables standardized measurement across studies and practical privacy assessment in real-world LLM applications.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, 3396 Fifth Avenue, Pittsburgh, PA, 15213, United States, 1 412-383-6653.
Cite
APA
Zhou, L., Wang, H., Xu, Y., Setiawan, I. M. A., Hu, H., Saptono, A., Choi, Y. K., & Parmanto, B. (2026). Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications. JMIR Human Factors, 13, e102910-e102910. https://doi.org/10.2196/102910
Vancouver
Zhou L, Wang H, Xu Y, Setiawan IMA, Hu H, Saptono A, et al. Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications. JMIR Human Factors. 2026;13:e102910-e102910. doi:10.2196/102910
BibTeX
@article{Zhou2026,
author = {Zhou, Leming and Wang, Hongtao and Xu, Yi and Setiawan, I Made Agus and Hu, Haomin and Saptono, Andi and Choi, Yong K and Parmanto, Bambang},
title = {Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications},
journal = {JMIR Human Factors},
year = {2026},
volume = {13},
pages = {e102910--e102910},
doi = {10.2196/102910}
}
EUVIMED