The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports
Systems · 11.10.2025 · 13 (10) · 899
DOI: 10.3390/systems13100899 ↗Kurz gefasst
Fachartikel in der Zeitschrift Systems, online erschienen am 11.10.2025. Verfasst von Yue Wu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 82 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
To explore the potential of Large Language Models (LLMs) as AI Annotators in the domain of sustainability reporting, this study establishes a systematic evaluation methodology. We use the specific case of European football clubs, quantifying their sustainability reports based on the sport Positive matrix as a benchmark to compare the performance of three state-of-the-art models (i.e., GPT-4o, Qwen-2-72b-instruct, and Llama-3-70b-instruct) against human expert scores. The evaluation is benchmarked on dimensions including accuracy, mean absolute error (MAE), and hallucination rates. The results indicate that GPT-4o is the top performer, yet its average accuracy of approximately 56% shows it cannot fully replace human experts at present. The study also reveals significant issues with overconfidence and factual hallucinations in models like Qwen-2-72b-instructon. Critically, we find that by implementing further data processing, specifically a Chain-of-Verification (CoVe) self-correction method, GPT-4o’s initial hallucination rate is successfully reduced from 16% to 10%, while accuracy improved to 58%. In conclusion, while LLMs demonstrate immense potential to streamline and democratize sustainability ratings, inherent risks like hallucinations remain a primary obstacle. Adopting verification strategies such as CoVe is a crucial pathway to enhancing model reliability and advancing their effective application in this field.
Zitieren
APA
Wu, Y., Hu, P., & Wang, D. D. (2025). The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports. Systems, 13(10), 899. https://doi.org/10.3390/systems13100899
Vancouver
Wu Y, Hu P, Wang DD. The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports. Systems. 2025;13(10):899. doi:10.3390/systems13100899
BibTeX
@article{Wu2025,
author = {Wu, Yue and Hu, Peng and Wang, Derek D.},
title = {The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports},
journal = {Systems},
year = {2025},
volume = {13},
number = {10},
pages = {899},
doi = {10.3390/systems13100899}
}
EUVIMED