Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study
Journal of Medical Internet Research · 21.08.2024 · 26 · e56500
DOI: 10.2196/56500 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 21.08.2024. Verfasst von Kevin Danis Li und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, University of California San Francisco, San Francisco, CA, United States. Hauptthemen laut MEDLINE: Qualitative Research. Die Arbeit wurde bisher 59-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models including GPT-4 (OpenAI) have opened new avenues in health care and qualitative research. Traditional qualitative methods are time-consuming and require expertise to capture nuance. Although large language models have demonstrated enhanced contextual understanding and inferencing compared with traditional natural language processing, their performance in qualitative analysis versus that of humans remains unexplored.
Objective
We evaluated the effectiveness of GPT-4 versus human researchers in qualitative analysis of interviews with patients with adult-acquired buried penis (AABP).
Methods
Qualitative data were obtained from semistructured interviews with 20 patients with AABP. Human analysis involved a structured 3-stage process—initial observations, line-by-line coding, and consensus discussions to refine themes. In contrast, artificial intelligence (AI) analysis with GPT-4 underwent two phases: (1) a naïve phase, where GPT-4 outputs were independently evaluated by a blinded reviewer to identify themes and subthemes and (2) a comparison phase, where AI-generated themes were compared with human-identified themes to assess agreement. We used a general qualitative description approach.
Results
The study population (N=20) comprised predominantly White (17/20, 85%), married (12/20, 60%), heterosexual (19/20, 95%) men, with a mean age of 58.8 years and BMI of 41.1 kg/m2. Human qualitative analysis identified “urinary issues” in 95% (19/20) and GPT-4 in 75% (15/20) of interviews, with the subtheme “spray or stream” noted in 60% (12/20) and 35% (7/20), respectively. “Sexual issues” were prominent (19/20, 95% humans vs 16/20, 80% GPT-4), although humans identified a wider range of subthemes, including “pain with sex or masturbation” (7/20, 35%) and “difficulty with sex or masturbation” (4/20, 20%). Both analyses similarly highlighted “mental health issues” (11/20, 55%, both), although humans coded “depression” more frequently (10/20, 50% humans vs 4/20, 20% GPT-4). Humans frequently cited “issues using public restrooms” (12/20, 60%) as impacting social life, whereas GPT-4 emphasized “struggles with romantic relationships” (9/20, 45%). “Hygiene issues” were consistently recognized (14/20, 70% humans vs 13/20, 65% GPT-4). Humans uniquely identified “contributing factors” as a theme in all interviews. There was moderate agreement between human and GPT-4 coding (κ=0.401). Reliability assessments of GPT-4’s analyses showed consistent coding for themes including “body image struggles,” “chronic pain” (10/10, 100%), and “depression” (9/10, 90%). Other themes like “motivation for surgery” and “weight challenges” were reliably coded (8/10, 80%), while less frequent themes were variably identified across multiple iterations.
Conclusions
Large language models including GPT-4 can effectively identify key themes in analyzing qualitative health care data, showing moderate agreement with human analysis. While human analysis provided a richer diversity of subthemes, the consistency of AI suggests its use as a complementary tool in qualitative research. With AI rapidly advancing, future studies should iterate analyses and circumvent token limitations by segmenting data, furthering the breadth and depth of large language model–driven qualitative analyses.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, University of California San Francisco, San Francisco, CA, United States.
- Department of Epidemiology and Biostatistics, University of California San Francisco, San Francisco, CA, United States.
- Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, CA, United States.
- Division of General Internal Medicine, Department of Medicine, University of California San Francisco, San Francisco, CA, United States.
- Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.
ORCID-Profile: Kevin Danis Li, Adrian M Fernandez, Rachel Schwartz, Natalie Rios, Marvin Nathaniel Carlisle, Gregory M Amend, Hiren V Patel, Benjamin N Breyer
Zitieren
APA
Li, K. D., Fernandez, A. M., Schwartz, R., Rios, N., Carlisle, M. N., Amend, G. M., Patel, H. V., & Breyer, B. N. (2024). Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study. Journal of Medical Internet Research, 26, e56500. https://doi.org/10.2196/56500
Vancouver
Li KD, Fernandez AM, Schwartz R, Rios N, Carlisle MN, Amend GM, et al. Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study. Journal of Medical Internet Research. 2024;26:e56500. doi:10.2196/56500
BibTeX
@article{Li2024,
author = {Li, Kevin Danis and Fernandez, Adrian M and Schwartz, Rachel and Rios, Natalie and Carlisle, Marvin Nathaniel and Amend, Gregory M and Patel, Hiren V and Breyer, Benjamin N},
title = {Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e56500},
doi = {10.2196/56500}
}
EUVIMED