Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts
PLOS Digital Health · 03.04.2026 · 5 (4) · e0001189
DOI: 10.1371/journal.pdig.0001189 ↗Kurz gefasst
Mixed-Methods-Studie in der Zeitschrift PLOS Digital Health, online erschienen am 03.04.2026. Verfasst von Callum Hill und 6 weiteren Autor:innen. Beteiligte Einrichtung: Primary Care Research Centre, University of Southampton, Southampton, United Kingdom. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Institute for Health and Care Research.
Abstract (Originalsprache)
Large language models (LLMs) are increasingly used for qualitative thematic analysis, yet evidence on their performance in analysing focus-group data, where polyvocality and context complicate coding, remains limited. Given the increasing role of such models in thematic analysis, there is a need for methodological frameworks that enable systematic, metric-based comparisons between human and model-based analyses. We conducted a blinded mixed-methods comparison of two general-purpose LLMs (ChatGPT-5 and Claude 4 Sonnet), an LLM-based qualitative coding application (QualiGPT), and blinded human analysts on an in-person focus-group transcript informing an AI-enabled digital health proposal. We evaluated deductive coding using a 10-code, 6-theme codebook against an expert consensus adjudication; inductive coding with a structured Likert-scale comparison to a reference-standard set of inductive themes generated by expert consensus; and manual quote verification of LLM segments to define LLM hallucination (evidence absent or non-supportive) and error rate (including partial matches and speaker-coded segments). During deductive coding against an expert consensus adjudication, large language models yielded a mean agreement of 93.5% (95% CI 92.5–94.5) with κ = 0.34 (95% CI 0.26–0.40); blinded human coders achieved 92.7% (95% CI 91.6–93.9) agreement with κ = 0.34 (95% CI 0.26–0.41). Mean Gwet’s AC1 was 0.92 (95% CI 0.90–0.93) for the blinded human analysis, and 0.93 (95% CI 0.92–0.94) for the LLM-assisted deductive analysis, reflecting high agreement despite the low overall code prevalence (7.8%, SD = 3.2%). Only one model achieved non-inferiority in inductive analysis of the transcript ( p = 0.043). The strict hallucination rate in inductive analysis was 1.2% (SD = 2.1%). LLMs were non-inferior to human analysts for deductive coding of the focus-group data, with variable performance in inductive analysis. Low hallucination but significant comprehensive error rates indicate that LLMs can augment qualitative analysis but require human verification.
Autor:innen und Institutionen
- Primary Care Research Centre, University of Southampton, Southampton, United Kingdom.
ORCID-Profile: Callum Hill, Arun Dahil, Cameron Kumar Pinn
Förderung
- National Institute for Health and Care Research
Zitieren
APA
Hill, C., Dahil, A., Simpson, G., Hardisty, D., Keast, J., Pinn, C. K., & Dambha-Miller, H. (2026). Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts. PLOS Digital Health, 5(4), e0001189. https://doi.org/10.1371/journal.pdig.0001189
Vancouver
Hill C, Dahil A, Simpson G, Hardisty D, Keast J, Pinn CK, et al. Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts. PLOS Digital Health. 2026;5(4):e0001189. doi:10.1371/journal.pdig.0001189
BibTeX
@article{Hill2026,
author = {Hill, Callum and Dahil, Arun and Simpson, Glenn and Hardisty, David and Keast, Jacob and Pinn, Cameron Kumar and Dambha-Miller, Hajira},
title = {Large language models for thematic analysis in healthcare research: A blinded mixed-methods comparison with human analysts},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {4},
pages = {e0001189},
doi = {10.1371/journal.pdig.0001189}
}
EUVIMED