EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis

Stavros Doropoulos, Elisavet Karapalidou, Polychronis Charitidis, Sophia Karakeva, Stavros Vologiannidis

Applied Sciences · 20.07.2025 · 15 (14) · 8059

DOI: 10.3390/app15148059 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.07.2025. Verfasst von Stavros Doropoulos und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The vast volume of media content, combined with the costs of manual annotation, challenges scalable codebook analysis and risks reducing decision-making accuracy. This study evaluates the effectiveness of large language models (LLMs) and multi-agent teams in structured media content analysis based on codebook-driven annotation. We construct a dataset of 200 news articles on U.S. tariff policies, manually annotated using a 26-question codebook encompassing 122 distinct codes, to establish a rigorous ground truth. Seven state-of-the-art LLMs, spanning low- to high-capacity tiers, are assessed under a unified zero-shot prompting framework incorporating role-based instructions and schema-constrained outputs. Experimental results show weighted global F1-scores between 0.636 and 0.822, with Claude-3-7-Sonnet achieving the highest direct-prompt performance. To examine the potential of agentic orchestration, we propose and develop a multi-agent system using Meta’s Llama 4 Maverick, incorporating expert role profiling, shared memory, and coordinated planning. This architecture improves the overall F1-score over the direct prompting baseline from 0.757 to 0.805 and demonstrates consistent gains across binary, categorical, and multi-label tasks, approaching commercial-level accuracy while maintaining a favorable cost–performance profile. These findings highlight the viability of LLMs, both in direct and agentic configurations, for automating structured content analysis.

Zitieren

APA

Doropoulos, S., Karapalidou, E., Charitidis, P., Karakeva, S., & Vologiannidis, S. (2025). Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis. Applied Sciences, 15(14), 8059. https://doi.org/10.3390/app15148059

Vancouver

Doropoulos S, Karapalidou E, Charitidis P, Karakeva S, Vologiannidis S. Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis. Applied Sciences. 2025;15(14):8059. doi:10.3390/app15148059

BibTeX
@article{Doropoulos2025,
  author = {Doropoulos, Stavros and Karapalidou, Elisavet and Charitidis, Polychronis and Karakeva, Sophia and Vologiannidis, Stavros},
  title = {Beyond Manual Media Coding: Evaluating Large Language Models and Agents for News Content Analysis},
  journal = {Applied Sciences},
  year = {2025},
  volume = {15},
  number = {14},
  pages = {8059},
  doi = {10.3390/app15148059}
}
Kostenlos starten →