EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications

Leming Zhou, Hongtao Wang, Yi Xu, I Made Agus Setiawan, Haomin Hu, Andi Saptono, Yong K Choi, Bambang Parmanto

JMIR Human Factors · 21.10.2026 · 13 · e102910-e102910

DOI: 10.2196/102910 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Human Factors, online erschienen am 21.10.2026. Verfasst von Leming Zhou und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, 3396 Fifth Avenue, Pittsburgh, P…. Hauptthemen laut MEDLINE: Large Language Models, Psychometrics, Privacy. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Privacy concerns represent a primary barrier to the adoption of large language model (LLM) applications, particularly in sensitive domains such as health care. Although a growing body of survey-based research has examined user privacy concerns with LLMs, no standardized instrument exists for measuring this construct.

Objective

This study aimed to develop and validate a concise, standardized questionnaire for assessing user privacy concerns with LLM-based systems, suitable for use by researchers and practitioners such as AI application developers and health care professionals.

Methods

A literature review was conducted to identify privacy-related domains relevant to LLM use. A draft questionnaire was developed and subjected to content validity review by 8 experts. The finalized instrument was administered via Qualtrics to adults aged 18 years or older who were aware of LLM-based applications, recruited through the University of Pittsburgh, LinkedIn, and the Pitt+Me research participant registry between April 2025 and July 2026. Internal consistency reliability and factor structure were assessed, with factor structure examined in a split-sample design, in which exploratory factor analysis (EFA) was conducted on a random one-third of the sample and confirmatory factor analysis (CFA) on the holdout two-thirds.

Results

All questionnaire items achieved acceptable content validity (item-level content validity index >0.80; average scale-level content validity index=0.964). A total of 603 responses were collected. After preprocessing, the final sample comprised 361 participants. Most participants were younger than 35 years of age (198/361, 54.8%), female (230/361, 63.7%), and White (285/361, 78.9%); 88.1% (318/361) held at least a bachelor’s degree. The 6-item questionnaire showed high internal consistency (Cronbach α=0.90; McDonald ω=0.90). EFA (n=121) supported a single dominant factor with all standardized loadings ≥0.70. CFA (n=240) showed excellent fit (comparative fit index=0.996; Tucker-Lewis index=0.993; root mean square error of approximation=0.046; standardized root mean square residual=0.057). A 2-factor model separating the transparency- and control-related items fit significantly better than the 1-factor model (Δ χ 2 1 =8.9; P =.003), but the 2 factors were correlated at r =0.83, indicating closely related facets. Because the Fornell-Larcker criterion was met for the transparency and control subscale but not for data use and risk salience, the 6-item total score remains the recommended primary metric.

Conclusions

The LLM privacy concern assessment is a concise, psychometrically sound 6-item instrument that measures privacy concern with LLM-based systems via 2 correlated subdimensions (transparency and control; data use and risk salience) summarized by a single total score. It enables standardized measurement across studies and practical privacy assessment in real-world LLM applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPsychometricsPrivacyHumansSurveys and QuestionnairesReproducibility of ResultsFemaleAdultMaleMiddle Aged

Stichwörter der Autor:innen

large language modelsprivacypsychometric analysisquestionnairescale validation

Autor:innen und Institutionen

  • Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, 3396 Fifth Avenue, Pittsburgh, PA, 15213, United States, 1 412-383-6653.

Zitieren

APA

Zhou, L., Wang, H., Xu, Y., Setiawan, I. M. A., Hu, H., Saptono, A., Choi, Y. K., & Parmanto, B. (2026). Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications. JMIR Human Factors, 13, e102910-e102910. https://doi.org/10.2196/102910

Vancouver

Zhou L, Wang H, Xu Y, Setiawan IMA, Hu H, Saptono A, et al. Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications. JMIR Human Factors. 2026;13:e102910-e102910. doi:10.2196/102910

BibTeX
@article{Zhou2026,
  author = {Zhou, Leming and Wang, Hongtao and Xu, Yi and Setiawan, I Made Agus and Hu, Haomin and Saptono, Andi and Choi, Yong K and Parmanto, Bambang},
  title = {Development and Validation of the Large Language Model Privacy Concern Assessment Questionnaire: Psychometric Evaluation Study on a Brief Instrument for Assessing User Privacy Concerns With Large Language Model Applications},
  journal = {JMIR Human Factors},
  year = {2026},
  volume = {13},
  pages = {e102910--e102910},
  doi = {10.2196/102910}
}
Kostenlos starten →