EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Using Large Language Models for Qualitative Analysis can Introduce Serious Bias

Julian Ashwin, Aditya Chhabra, Vijayendra Rao

Sociological Methods & Research · 2026-08 · 55 (3) · 795-839

DOI: 10.1177/00491241251338246 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sociological Methods & Research, online erschienen am 27.05.2025. Verfasst von Julian Ashwin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: World Bank’s Knowledge for Change Program, World Bank-UNHCR Joint Data Center on Forced Displacement.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are quickly becoming ubiquitous, but their implications for social science research are not yet well understood. We ask whether LLMs can help code and analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with Rohingya refugees and their Bengali hosts in Bangladesh. We find that using LLMs to annotate and code text can introduce bias that can lead to misleading inferences. By bias we mean that the errors that LLMs make in coding interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human codes leads to less measurement error and bias than LLM annotations. Given that high quality codes are necessary in order to assess whether an LLM introduces bias, we argue that it may be preferable to train a bespoke model on a subset of transcripts coded by trained sociologists rather than use an LLM.

Förderung

  • World Bank’s Knowledge for Change Program
  • World Bank-UNHCR Joint Data Center on Forced Displacement

Zitieren

APA

Ashwin, J., Chhabra, A., & Rao, V. (2026). Using Large Language Models for Qualitative Analysis can Introduce Serious Bias. Sociological Methods & Research, 55(3), 795-839. https://doi.org/10.1177/00491241251338246

Vancouver

Ashwin J, Chhabra A, Rao V. Using Large Language Models for Qualitative Analysis can Introduce Serious Bias. Sociological Methods & Research. 2026;55(3):795-839. doi:10.1177/00491241251338246

BibTeX
@article{Ashwin2026,
  author = {Ashwin, Julian and Chhabra, Aditya and Rao, Vijayendra},
  title = {Using Large Language Models for Qualitative Analysis can Introduce Serious Bias},
  journal = {Sociological Methods & Research},
  year = {2026},
  volume = {55},
  number = {3},
  pages = {795--839},
  doi = {10.1177/00491241251338246}
}
Kostenlos starten →