EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting

Md Hamid Hosen iD, Sadia Nawar, Rituparna Chowdhury, Altaf Uddin, Riadul Islam Rabbi, Khair Razlan Othman iD, Nurul Karim Symon

PLOS One · 06.11.2026 · 21 (11) · e0361368

DOI: 10.1371/journal.pone.0361368 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 06.11.2026. Verfasst von Md Hamid Hosen und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, East Delta University, Chattogram, Bangladesh. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have advanced multilingual NLP significantly, yet their effectiveness for Bengali, a globally important but low-resource language, remains understudied due to limited linguistic resources and the absence of a standardized evaluation benchmark. This is due to the unavailability of large linguistic resources and the lack of a standard benchmarking framework, which has yet to be established. This study aims to address the underrepresentation of Bengali text classification by leading a systematic evaluation. It involves assessing multiple open-weight LLMs on Bengali text classification tasks and observing their limitations and regular error patterns. The procedure consists of six open-weight LLMs, which include LLaMA-3.2-3B, Mistral-V3-7B, DeepSeek-R1-8B, Phi-4-14B, Gemma-2-27B, and Qwen-2.5-72B, and are evaluated using four critical text classification tasks in Bengali, namely sentiment analysis, emotion recognition, hate-speech detection, and fake-news classification. Furthermore, two prompting methods are compared in the context of zero-shot and few-shot prompting using both the vLLM inference engine and Hugging Face Transformers pipelines to investigate the adaptability of the models in resource-limited settings. Across all four tasks, the two largest models (Qwen-2.5-72B and Gemma-2-27B) obtained the highest scores, although the ranking was not strictly monotonic in parameter count and, because each scale is represented by a single model family, the design cannot attribute these differences to model size alone. When training in zero-shot, Qwen-2.5-72B showed the best performance at 79.90% accuracy and 79.88% Macro-F1 score, showing amazing accuracy on hate-speech detection. In the case of fake news, Gemma-2-27B performed best at 84.45% accuracy and 84.33% Macro-F1 score. With a few-shot set-up, Gemma-2-27B and Qwen-2.5-72B demonstrated a high level of performance in terms of Macro-F1 score, reaching 84.17% for fake-news detection and 66.80% for emotion recognition, respectively. Paired multi-model tests (Cochran’s Q with Holm-corrected McNemar post-hoc comparisons) confirmed significant differences among the six models across all tasks and prompting settings, whereas the effect of five-shot prompting, obtained with a single fixed demonstration set, was significant for only 4 of 24 model–task combinations after correction and never for the three largest models. The findings characterise benchmark classification performance under the evaluated settings rather than real-world effectiveness, and indicate that model scale, multilingual pre-training, and in-context demonstrations are jointly associated with, but cannot be causally isolated as drivers of, performance on Bengali text classification.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingLarge Language ModelsHumansLinguistics

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, East Delta University, Chattogram, Bangladesh.
  • Faculty of Engineering and Technology, Multimedia University, Melaka, Malaysia.
  • Centre for Advanced Analytics, COE for Artificial Intelligence, Faculty of Engineering and Technology (FET), Multimedia University, Melaka, Malaysia.
  • Department of Computer Science and Engineering, International Islamic University Chittagong, Chattogram, Bangladesh.

ORCID-Profile: Md Hamid Hosen, Khair Razlan Othman

Zitieren

APA

Hosen, M. H., Nawar, S., Chowdhury, R., Uddin, A., Rabbi, R. I., Othman, K. R., & Symon, N. K. (2026). Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting. PLOS One, 21(11), e0361368. https://doi.org/10.1371/journal.pone.0361368

Vancouver

Hosen MH, Nawar S, Chowdhury R, Uddin A, Rabbi RI, Othman KR, et al. Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting. PLOS One. 2026;21(11):e0361368. doi:10.1371/journal.pone.0361368

BibTeX
@article{Hosen2026,
  author = {Hosen, Md Hamid and Nawar, Sadia and Chowdhury, Rituparna and Uddin, Altaf and Rabbi, Riadul Islam and Othman, Khair Razlan and Symon, Nurul Karim},
  title = {Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {11},
  pages = {e0361368},
  doi = {10.1371/journal.pone.0361368}
}
Kostenlos starten →