EUVIMEDOpdag EUVIMED
VergleichsstudieOpen access

Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study

Kevin Danis Li iD, Adrian M Fernandez iD, Rachel Schwartz iD, Natalie Rios iD, Marvin Nathaniel Carlisle iD, Gregory M Amend iD, Hiren V Patel iD, Benjamin N Breyer iD

Journal of Medical Internet Research · 2024-08-21 · 26 · e56500

DOI: 10.2196/56500 ↗

At a glance

Vergleichsstudie in Journal of Medical Internet Research, published online on 2024-08-21. Written by Kevin Danis Li and 7 co-authors. Institution: Department of Urology, University of California San Francisco, San Francisco, CA, United States. Major MEDLINE topics: Qualitative Research. Cited 59 times so far (Crossref); 43 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Large language models including GPT-4 (OpenAI) have opened new avenues in health care and qualitative research. Traditional qualitative methods are time-consuming and require expertise to capture nuance. Although large language models have demonstrated enhanced contextual understanding and inferencing compared with traditional natural language processing, their performance in qualitative analysis versus that of humans remains unexplored.

Objective

We evaluated the effectiveness of GPT-4 versus human researchers in qualitative analysis of interviews with patients with adult-acquired buried penis (AABP).

Methods

Qualitative data were obtained from semistructured interviews with 20 patients with AABP. Human analysis involved a structured 3-stage process—initial observations, line-by-line coding, and consensus discussions to refine themes. In contrast, artificial intelligence (AI) analysis with GPT-4 underwent two phases: (1) a naïve phase, where GPT-4 outputs were independently evaluated by a blinded reviewer to identify themes and subthemes and (2) a comparison phase, where AI-generated themes were compared with human-identified themes to assess agreement. We used a general qualitative description approach.

Results

The study population (N=20) comprised predominantly White (17/20, 85%), married (12/20, 60%), heterosexual (19/20, 95%) men, with a mean age of 58.8 years and BMI of 41.1 kg/m2. Human qualitative analysis identified “urinary issues” in 95% (19/20) and GPT-4 in 75% (15/20) of interviews, with the subtheme “spray or stream” noted in 60% (12/20) and 35% (7/20), respectively. “Sexual issues” were prominent (19/20, 95% humans vs 16/20, 80% GPT-4), although humans identified a wider range of subthemes, including “pain with sex or masturbation” (7/20, 35%) and “difficulty with sex or masturbation” (4/20, 20%). Both analyses similarly highlighted “mental health issues” (11/20, 55%, both), although humans coded “depression” more frequently (10/20, 50% humans vs 4/20, 20% GPT-4). Humans frequently cited “issues using public restrooms” (12/20, 60%) as impacting social life, whereas GPT-4 emphasized “struggles with romantic relationships” (9/20, 45%). “Hygiene issues” were consistently recognized (14/20, 70% humans vs 13/20, 65% GPT-4). Humans uniquely identified “contributing factors” as a theme in all interviews. There was moderate agreement between human and GPT-4 coding (κ=0.401). Reliability assessments of GPT-4’s analyses showed consistent coding for themes including “body image struggles,” “chronic pain” (10/10, 100%), and “depression” (9/10, 90%). Other themes like “motivation for surgery” and “weight challenges” were reliably coded (8/10, 80%), while less frequent themes were variably identified across multiple iterations.

Conclusions

Large language models including GPT-4 can effectively identify key themes in analyzing qualitative health care data, showing moderate agreement with human analysis. While human analysis provided a richer diversity of subthemes, the consistency of AI suggests its use as a complementary tool in qualitative research. With AI rapidly advancing, future studies should iterate analyses and circumvent token limitations by segmenting data, furthering the breadth and depth of large language model–driven qualitative analyses.

Subjects

MeSH terms (MEDLINE)

Qualitative ResearchHumansMaleAdultMiddle AgedData AnalysisResearch PersonnelAged

Author keywords

ChatGPTartificial intelligenceburied peniscontent analysislarge language modelsqualitative analysisqualitative descriptionqualitative interviewsurology

Authors and institutions

  • Department of Urology, University of California San Francisco, San Francisco, CA, United States.
  • Department of Epidemiology and Biostatistics, University of California San Francisco, San Francisco, CA, United States.
  • Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, CA, United States.
  • Division of General Internal Medicine, Department of Medicine, University of California San Francisco, San Francisco, CA, United States.
  • Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, United States.

ORCID profiles: Kevin Danis Li, Adrian M Fernandez, Rachel Schwartz, Natalie Rios, Marvin Nathaniel Carlisle, Gregory M Amend, Hiren V Patel, Benjamin N Breyer

Cite

APA

Li, K. D., Fernandez, A. M., Schwartz, R., Rios, N., Carlisle, M. N., Amend, G. M., Patel, H. V., & Breyer, B. N. (2024). Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study. Journal of Medical Internet Research, 26, e56500. https://doi.org/10.2196/56500

Vancouver

Li KD, Fernandez AM, Schwartz R, Rios N, Carlisle MN, Amend GM, et al. Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study. Journal of Medical Internet Research. 2024;26:e56500. doi:10.2196/56500

BibTeX
@article{Li2024,
  author = {Li, Kevin Danis and Fernandez, Adrian M and Schwartz, Rachel and Rios, Natalie and Carlisle, Marvin Nathaniel and Amend, Gregory M and Patel, Hiren V and Breyer, Benjamin N},
  title = {Comparing GPT-4 and Human Researchers in Health Care Data Analysis: Qualitative Description Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e56500},
  doi = {10.2196/56500}
}
Start for free →