EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale

Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, Marta Miori, Phillip Schmedeman

Research & Politics · 2024-01 · 11 (1)

DOI: 10.1177/20531680241231468 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research & Politics, online erschienen am 12.02.2024. Verfasst von Jonathan Mellon und 5 weiteren Autor:innen. Die Arbeit wurde bisher 55-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Economic and Social Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Can artificial intelligence accurately label open-text survey responses? We compare the accuracy of six large language models (LLMs) using a few-shot approach, three supervised learning algorithms (SVM, DistilRoBERTa, and a neural network trained on BERT embeddings), and a second human coder on the task of categorizing “most important issue” responses from the British Election Study Internet Panel into 50 categories. For the scenario where a researcher lacks existing training data, the accuracy of the highest-performing LLM (Claude-1.3: 93.9%) neared human performance (94.7%) and exceeded the highest-performing supervised approach trained on 1000 randomly sampled cases (neural network: 93.5%). In a scenario where previous data has been labeled but a researcher wants to label novel text, the best LLM’s (Claude-1.3: 80.9%) few-shot performance is only slightly behind the human (88.6%) and exceeds the best supervised model trained on 576,000 cases (DistilRoBERTa: 77.8%). PaLM-2, Llama-2, and the SVM all performed substantially worse than the best LLMs and supervised models across all metrics and scenarios. Our results suggest that LLMs may allow for greater use of open-ended survey questions in the future.

Förderung

  • Economic and Social Research Council

Zitieren

APA

Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., & Schmedeman, P. (2024). Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale. Research & Politics, 11(1). https://doi.org/10.1177/20531680241231468

Vancouver

Mellon J, Bailey J, Scott R, Breckwoldt J, Miori M, Schmedeman P. Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale. Research & Politics. 2024;11(1). doi:10.1177/20531680241231468

BibTeX
@article{Mellon2024,
  author = {Mellon, Jonathan and Bailey, Jack and Scott, Ralph and Breckwoldt, James and Miori, Marta and Schmedeman, Phillip},
  title = {Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale},
  journal = {Research & Politics},
  year = {2024},
  volume = {11},
  number = {1},
  doi = {10.1177/20531680241231468}
}
Kostenlos starten →