Leveraging large language models for thematic analysis: a case study in the charity sector
AI & SOCIETY · 2026-01 · 41 (1) · 731-748
DOI: 10.1007/s00146-025-02487-4 ↗Kurz gefasst
Fachartikel in der Zeitschrift AI & SOCIETY, online erschienen am 17.08.2025. Verfasst von Chuanchi Wen und 3 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
This study explores how large language models (LLMs) can support deductive and inductive thematic coding in real-life contexts, balancing AI-driven efficiency with essential human oversight. Using three datasets from Tearfund, a UK-based Christian charity, we propose a dual-role human–LLM collaborative framework where the LLM functions as an initial annotator and a validator. In the deductive phase, GPT-4o and GPT-4o-mini were compared against human coders. GPT-4o achieved a substantial agreement in multi-label thematic categorization (κ = 0.61–0.65), while GPT-4o-mini showed a moderate agreement (κ = 0.41–0.58). Both models excelled in sentiment analysis (κ = 0.91–0.95), but struggled with evaluating evidence of impact due to contextual complexity (κ ≤ 0.01). GPT-4o-mini exhibited greater output variability and instability than GPT-4o, but benefited more from few-shot learning to mitigate hallucinations. In the inductive phase, GPT-4o demonstrated a strong semantic alignment with human-generated themes (cosine similarity = 0.76–0.79) though its tendency toward broad themes required human refinement. Despite their potential to streamline thematic analysis, LLMs also pose limitations and implementation challenges, including inconsistencies in excerpt extraction (precision = 0.41, recall = 0.53) and the trade-off between the time saved in coding and the time required for human validation. To facilitate practical implementation, we provide reusable prompt templates for four stages: context, instructions, data processing, and verification. Our findings underline the indispensable role of human expertise—from prompt engineering and managing hallucinations to final verification—to ensure accurate and trustworthy AI-assisted analyses. While LLMs can enhance qualitative analysis, their full potential is only realized under skilled human guidance.
Zitieren
APA
Wen, C., Clough, P., Paton, R., & Middleton, R. (2026). Leveraging large language models for thematic analysis: a case study in the charity sector. AI & SOCIETY, 41(1), 731-748. https://doi.org/10.1007/s00146-025-02487-4
Vancouver
Wen C, Clough P, Paton R, Middleton R. Leveraging large language models for thematic analysis: a case study in the charity sector. AI & SOCIETY. 2026;41(1):731-748. doi:10.1007/s00146-025-02487-4
BibTeX
@article{Wen2026,
author = {Wen, Chuanchi and Clough, Paul and Paton, Rachel and Middleton, Rebecca},
title = {Leveraging large language models for thematic analysis: a case study in the charity sector},
journal = {AI & SOCIETY},
year = {2026},
volume = {41},
number = {1},
pages = {731--748},
doi = {10.1007/s00146-025-02487-4}
}
EUVIMED