EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking

Aytug Onan, Arbi Haza Nasution, Tugba Celikten

IEEE Access · 2025 · 13 · 211620-211644

DOI: 10.1109/access.2025.3643829 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von Aytug Onan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universitas Islam Riau.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper introduces the Mixture-of-Agents (MoA) framework, a structured approach for improving the reliability of large language model (LLM)-based text annotation in low-resource NLP contexts. MoA employs coordinated agent interactions to enhance agreement, interpretability, and robustness without manual supervision. Evaluations on Turkish classification benchmarks demonstrate that MoA achieves up to 10-point improvements in macro-F1 over single-model baselines and significantly increases inter-agent consistency. Additionally, three novel reliability metrics—Conflict Rate (CR), Ambiguity Resolution Success Rate (ARSR), and Refinement Correction Rate (RCR)—are proposed to quantify annotation stability and correction dynamics. The results indicate that multi-agent coordination can substantially improve label quality, offering a scalable pathway toward trustworthy annotation in low-resource and cross-domain applications. The framework is language-agnostic and adaptable to other low-resource contexts beyond Turkish, including morphologically rich or typologically diverse languages such as Indonesian, Urdu, and Swahili. These findings highlight the scalability of MoA as a generalizable solution for multilingual and cross-domain annotation.

Förderung

  • Universitas Islam Riau

Zitieren

APA

Onan, A., Nasution, A. H., & Celikten, T. (2025). Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking. IEEE Access, 13, 211620-211644. https://doi.org/10.1109/access.2025.3643829

Vancouver

Onan A, Nasution AH, Celikten T. Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking. IEEE Access. 2025;13:211620-211644. doi:10.1109/access.2025.3643829

BibTeX
@article{Onan2025,
  author = {Onan, Aytug and Nasution, Arbi Haza and Celikten, Tugba},
  title = {Toward Reliable Annotation in Low-Resource NLP: A Mixture of Agents Framework and Multi-LLM Benchmarking},
  journal = {IEEE Access},
  year = {2025},
  volume = {13},
  pages = {211620--211644},
  doi = {10.1109/access.2025.3643829}
}
Kostenlos starten →