Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning
Journal of Computational Social Science · 2025-02 · 8 (1)
DOI: 10.1007/s42001-024-00345-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computational Social Science, online erschienen am 18.12.2024. Verfasst von Meysam Alizadeh und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Political Science, University of Zurich, 8050 Zurich, Switzerland. Die Arbeit wurde bisher 85-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: H2020 European Research Council, University of Zurich.
Abstract (Originalsprache)
This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ - 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ - 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Political Science, University of Zurich, 8050 Zurich, Switzerland.
- Department of Computer Science, Institute for Fundamental Research, Tehran, Iran.
- Department of Computer Engineering, Allameh Tabataba'i University, Tehran, Iran.
- Department of Computer Science, Iran University of Science and Technology, Tehran, Iran.
ORCID-Profile: Meysam Alizadeh
Förderung
- H2020 European Research Council
- University of Zurich
Zitieren
APA
Alizadeh, M., Kubli, M., Samei, Z., Dehghani, S., Zahedivafa, M., Bermeo, J. D., Korobeynikova, M., & Gilardi, F. (2025). Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science, 8(1). https://doi.org/10.1007/s42001-024-00345-9
Vancouver
Alizadeh M, Kubli M, Samei Z, Dehghani S, Zahedivafa M, Bermeo JD, et al. Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning. Journal of Computational Social Science. 2025;8(1). doi:10.1007/s42001-024-00345-9
BibTeX
@article{Alizadeh2025,
author = {Alizadeh, Meysam and Kubli, Maël and Samei, Zeynab and Dehghani, Shirin and Zahedivafa, Mohammadmasiha and Bermeo, Juan D and Korobeynikova, Maria and Gilardi, Fabrizio},
title = {Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning},
journal = {Journal of Computational Social Science},
year = {2025},
volume = {8},
number = {1},
doi = {10.1007/s42001-024-00345-9}
}
EUVIMED