EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Promises and pitfalls of using LLMs to identify actor stances in political discourse

Viviane Walker iD, Mario Angst iD

PLOS One · 2025

DOI: 10.1371/journal.pone.0335547 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, 2025. Verfasst von Viviane Walker und 1 weiteren Autor:innen. Beteiligte Einrichtung: University of Zürich, Zürich, Switzerland. Hauptthemen laut MEDLINE: Politics, Language. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Digitalisierungsinitiative der Zürcher Hochschulen.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Empirical research in the social sciences is often interested in understanding actor stances; the positions that social actors take regarding normative statements in societal discourse. In automated text analysis applications, the classification task of stance detection remains challenging. Stance detection is especially difficult due to semantic challenges such as implicitness or missing context but also due to the general nature of the task. In this paper, we explore the potential of Large Language Models (LLMs) to enable stance detection in a generalized (non-domain, non-statement specific) form. Specifically, we test a variety of different general prompt chains for zero-shot stance classifications. Our evaluation data consists of textual data from a real-world empirical research project in the domain of sustainable urban transport. For 1710 German newspaper paragraphs, each containing an organizational entity, we annotated the stance of the entity toward one of five normative statements. A comparison of four publicly available LLMs show that they can achieve adequate performance. However, results heavily depend on the prompt chain method, LLM, and vary by statement. Our findings have implications for computational linguistics methodology and political discourse analysis, as they offer a deeper understanding of the strengths and weaknesses of LLMs in performing the complex semantic task of stance detection. We strongly emphasise the necessity of domain-specific evaluation data for evaluating LLMs, considering trade-offs between model complexity and performance, as well as honestly weighing drawbacks of LLM application against traditional, valid approaches, such as manually annotating representative text samples.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PoliticsLanguageHumansSemanticsModels, TheoreticalLinguistics

Autor:innen und Institutionen

  • University of Zürich, Zürich, Switzerland.

ORCID-Profile: Viviane Walker, Mario Angst

Förderung

  • Digitalisierungsinitiative der Zürcher Hochschulen

Zitieren

APA

Walker, V., & Angst, M. (2025). Promises and pitfalls of using LLMs to identify actor stances in political discourse. PLOS One. https://doi.org/10.1371/journal.pone.0335547

Vancouver

Walker V, Angst M. Promises and pitfalls of using LLMs to identify actor stances in political discourse. PLOS One. 2025. doi:10.1371/journal.pone.0335547

BibTeX
@article{Walker2025,
  author = {Walker, Viviane and Angst, Mario},
  title = {Promises and pitfalls of using LLMs to identify actor stances in political discourse},
  journal = {PLOS One},
  year = {2025},
  doi = {10.1371/journal.pone.0335547}
}
Kostenlos starten →