EUVIMEDEUVIMED entdecken
Qualitative StudieOpen Access

ChatGPT for Automated Qualitative Research: Content Analysis

Rimke Bijker iD, Stephanie S Merkouris iD, Nicki A Dowling iD, Simone N Rodda iD

Journal of Medical Internet Research · 25.07.2024 · 26 · e59050

DOI: 10.2196/59050 ↗

Kurz gefasst

Qualitative Studie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 25.07.2024. Verfasst von Rimke Bijker und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology and Neuroscience, Auckland University of Technology, Auckland, New Zealand. Hauptthemen laut MEDLINE: Qualitative Research, Artificial Intelligence. Die Arbeit wurde bisher 171-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Data analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis.

Objective

The aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption.

Methods

Inductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions.

Results

The precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified.

Conclusions

ChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Qualitative ResearchArtificial IntelligenceHumans

Stichwörter der Autor:innen

ChatGPTTheoretical Domains Frameworknatural language processingqualitative content analysis

Autor:innen und Institutionen

  • Department of Psychology and Neuroscience, Auckland University of Technology, Auckland, New Zealand.
  • School of Psychology, Deakin University, Burwood, Australia.

ORCID-Profile: Rimke Bijker, Stephanie S Merkouris, Nicki A Dowling, Simone N Rodda

Zitieren

APA

Bijker, R., Merkouris, S. S., Dowling, N. A., & Rodda, S. N. (2024). ChatGPT for Automated Qualitative Research: Content Analysis. Journal of Medical Internet Research, 26, e59050. https://doi.org/10.2196/59050

Vancouver

Bijker R, Merkouris SS, Dowling NA, Rodda SN. ChatGPT for Automated Qualitative Research: Content Analysis. Journal of Medical Internet Research. 2024;26:e59050. doi:10.2196/59050

BibTeX
@article{Bijker2024,
  author = {Bijker, Rimke and Merkouris, Stephanie S and Dowling, Nicki A and Rodda, Simone N},
  title = {ChatGPT for Automated Qualitative Research: Content Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e59050},
  doi = {10.2196/59050}
}
Kostenlos starten →