Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking
IEEE Access · 2025 · 13 · 211620-211644
DOI: 10.1109/access.2025.3643829 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von Aytug Onan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universitas Islam Riau.
Abstract (Originalsprache)
This paper introduces the Mixture-of-Agents (MoA) framework, a structured approach for improving the reliability of large language model (LLM)-based text annotation in low-resource NLP contexts. MoA employs coordinated agent interactions to enhance agreement, interpretability, and robustness without manual supervision. Evaluations on Turkish classification benchmarks demonstrate that MoA achieves up to 10-point improvements in macro-F1 over single-model baselines and significantly increases inter-agent consistency. Additionally, three novel reliability metrics—Conflict Rate (CR), Ambiguity Resolution Success Rate (ARSR), and Refinement Correction Rate (RCR)—are proposed to quantify annotation stability and correction dynamics. The results indicate that multi-agent coordination can substantially improve label quality, offering a scalable pathway toward trustworthy annotation in low-resource and cross-domain applications. The framework is language-agnostic and adaptable to other low-resource contexts beyond Turkish, including morphologically rich or typologically diverse languages such as Indonesian, Urdu, and Swahili. These findings highlight the scalability of MoA as a generalizable solution for multilingual and cross-domain annotation.
Förderung
- Universitas Islam Riau
Zitieren
APA
Onan, A., Nasution, A. H., & Celikten, T. (2025). Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking. IEEE Access, 13, 211620-211644. https://doi.org/10.1109/access.2025.3643829
Vancouver
Onan A, Nasution AH, Celikten T. Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking. IEEE Access. 2025;13:211620-211644. doi:10.1109/access.2025.3643829
BibTeX
@article{Onan2025,
author = {Onan, Aytug and Nasution, Arbi Haza and Celikten, Tugba},
title = {Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking},
journal = {IEEE Access},
year = {2025},
volume = {13},
pages = {211620--211644},
doi = {10.1109/access.2025.3643829}
}
EUVIMED