EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning

Meysam Alizadeh iD, Maël Kubli, Zeynab Samei, Shirin Dehghani, Mohammadmasiha Zahedivafa, Juan D Bermeo, Maria Korobeynikova, Fabrizio Gilardi

Journal of Computational Social Science · 2025-02 · 8 (1)

DOI: 10.1007/s42001-024-00345-9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Computational Social Science, online erschienen am 18.12.2024. Verfasst von Meysam Alizadeh und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Political Science, University of Zurich, 8050 Zurich, Switzerland. Die Arbeit wurde bisher 85-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: H2020 European Research Council, University of Zurich.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ - 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ - 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTFLANLLMsLLaMANLPOpen sourceText annotation

Autor:innen und Institutionen

  • Department of Political Science, University of Zurich, 8050 Zurich, Switzerland.
  • Department of Computer Science, Institute for Fundamental Research, Tehran, Iran.
  • Department of Computer Engineering, Allameh Tabataba'i University, Tehran, Iran.
  • Department of Computer Science, Iran University of Science and Technology, Tehran, Iran.

ORCID-Profile: Meysam Alizadeh

Förderung

  • H2020 European Research Council
  • University of Zurich

Zitieren

APA

Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2025). Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1). https://doi.org/10.1007/s42001-024-00345-9

Vancouver

Alizadeh M, Kubli M, Samei Z, Dehghani S, Zahedivafa M, Bermeo JD, et al. Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science. 2025;8(1). doi:10.1007/s42001-024-00345-9

BibTeX
@article{Alizadeh2025,
  author = {Alizadeh, Meysam and Kubli, Maël and Samei, Zeynab and Dehghani, Shirin and Zahedivafa, Mohammadmasiha and Bermeo, Juan D and Korobeynikova, Maria and Gilardi, Fabrizio},
  title = {Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning},
  journal = {Journal of Computational Social Science},
  year = {2025},
  volume = {8},
  number = {1},
  doi = {10.1007/s42001-024-00345-9}
}
Kostenlos starten →