EUVIMEDEUVIMED entdecken
Mixed-Methods-StudieOpen Access · CC BY 4.0

Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts

Callum Hill iD, Arun Dahil iD, Glenn Simpson, David Hardisty, Jacob Keast, Cameron Kumar Pinn iD, Hajira Dambha-Miller

PLOS Digital Health · 03.04.2026 · 5 (4) · e0001189

DOI: 10.1371/journal.pdig.0001189 ↗

Kurz gefasst

Mixed-Methods-Studie in der Zeitschrift PLOS Digital Health, online erschienen am 03.04.2026. Verfasst von Callum Hill und 6 weiteren Autor:innen. Beteiligte Einrichtung: Primary Care Research Centre, University of Southampton, Southampton, United Kingdom. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institute for Health and Care Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly used for qualitative thematic analysis, yet evidence on their performance in analysing focus-group data, where polyvocality and context complicate coding, remains limited. Given the increasing role of such models in thematic analysis, there is a need for methodological frameworks that enable systematic, metric-based comparisons between human and model-based analyses. We conducted a blinded mixed-methods comparison of two general-purpose LLMs (ChatGPT-5 and Claude 4 Sonnet), an LLM-based qualitative coding application (QualiGPT), and blinded human analysts on an in-person focus-group transcript informing an AI-enabled digital health proposal. We evaluated deductive coding using a 10-code, 6-theme codebook against an expert consensus adjudication; inductive coding with a structured Likert-scale comparison to a reference-standard set of inductive themes generated by expert consensus; and manual quote verification of LLM segments to define LLM hallucination (evidence absent or non-supportive) and error rate (including partial matches and speaker-coded segments). During deductive coding against an expert consensus adjudication, large language models yielded a mean agreement of 93.5% (95% CI 92.5–94.5) with κ = 0.34 (95% CI 0.26–0.40); blinded human coders achieved 92.7% (95% CI 91.6–93.9) agreement with κ = 0.34 (95% CI 0.26–0.41). Mean Gwet’s AC1 was 0.92 (95% CI 0.90–0.93) for the blinded human analysis, and 0.93 (95% CI 0.92–0.94) for the LLM-assisted deductive analysis, reflecting high agreement despite the low overall code prevalence (7.8%, SD = 3.2%). Only one model achieved non-inferiority in inductive analysis of the transcript ( p = 0.043). The strict hallucination rate in inductive analysis was 1.2% (SD = 2.1%). LLMs were non-inferior to human analysts for deductive coding of the focus-group data, with variable performance in inductive analysis. Low hallucination but significant comprehensive error rates indicate that LLMs can augment qualitative analysis but require human verification.

Autor:innen und Institutionen

  • Primary Care Research Centre, University of Southampton, Southampton, United Kingdom.

ORCID-Profile: Callum Hill, Arun Dahil, Cameron Kumar Pinn

Förderung

  • National Institute for Health and Care Research

Zitieren

APA

Hill, C., Dahil, A., Simpson, G., Hardisty, D., Keast, J., Pinn, C. K., & Dambha-Miller, H. (2026). Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts. PLOS Digital Health, 5(4), e0001189. https://doi.org/10.1371/journal.pdig.0001189

Vancouver

Hill C, Dahil A, Simpson G, Hardisty D, Keast J, Pinn CK, et al. Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts. PLOS Digital Health. 2026;5(4):e0001189. doi:10.1371/journal.pdig.0001189

BibTeX
@article{Hill2026,
  author = {Hill, Callum and Dahil, Arun and Simpson, Glenn and Hardisty, David and Keast, Jacob and Pinn, Cameron Kumar and Dambha-Miller, Hajira},
  title = {Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts},
  journal = {PLOS Digital Health},
  year = {2026},
  volume = {5},
  number = {4},
  pages = {e0001189},
  doi = {10.1371/journal.pdig.0001189}
}
Kostenlos starten →