EUVIMEDEUVIMED entdecken
Crossref

Evaluation of Monolingual and Multilingual Transformer Models for Nepali Headline Generation

Sunil Dahal, Katsuhide Fujita

ACM Transactions on Asian and Low-Resource Language Information Processing · 21.05.2026

DOI: 10.1145/3816146 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Asian and Low-Resource Language Information Processing, online erschienen am 21.05.2026. Verfasst von Sunil Dahal und 1 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Due to the scarcity of annotated data, NLP tools, and pre-trained models, developing reliable natural language generation (NLG) systems for low-resource languages continues to be a major issue. This paper presents a comprehensive evaluation of four Transformer-based architectures for the news headline generation in the Nepali, a morphologically rich and underrepresented language. We evaluated four models representing different architectural paradigms: mBART-large-cc25 and mT5-small (multilingual encoder–decoder models), NepBERTa (a monolingual encoder model adapted to a sequence-to-sequence architecture), and LLaMA-3.2-1B (a multilingual decoder-only model, not finetuned in Nepali). All models were fine-tuned on a custom-curated dataset (NEPHEAD) under a unified experimental setup to enable controlled comparison. Model performance was evaluated using both lexical and semantic automatic metrics, that include ROUGE, BLEU, METEOR, BERTScore, and SBERT-based similarity, along with human evaluation based on relevance, fluency, conciseness, accuracy, and engagement. Inter-annotator agreement using Spearman’s rank correlation coefficient ( ρ ) and Pearson correlation coefficient (r) was also ccalculated to assess the reliability of evaluation. The results show that multilingual encoder–decoder models, particularly mBART-large-cc25, achieve the best overall performance in both automatic and human evaluation. The mT5-small model performs competitively, while LLaMA-3.2-1B demonstrates strong adaptability despite the absence of explicit Nepali pretraining. In contrast, NepBERTa exhibits limited effectiveness due to architectural and tokenization constraints. These findings highlight the importance of pretraining objectives, model architecture, and tokenizer design in low-resource text generation and provide practical insight to develop reliable NLG systems in underrepresented languages.

Zitieren

APA

Dahal, S., & Fujita, K. (2026). Evaluation of Monolingual and Multilingual Transformer Models for Nepali Headline Generation. ACM Transactions on Asian and Low-Resource Language Information Processing. https://doi.org/10.1145/3816146

Vancouver

Dahal S, Fujita K. Evaluation of Monolingual and Multilingual Transformer Models for Nepali Headline Generation. ACM Transactions on Asian and Low-Resource Language Information Processing. 2026. doi:10.1145/3816146

BibTeX
@article{Dahal2026,
  author = {Dahal, Sunil and Fujita, Katsuhide},
  title = {Evaluation of Monolingual and Multilingual Transformer Models for Nepali Headline Generation},
  journal = {ACM Transactions on Asian and Low-Resource Language Information Processing},
  year = {2026},
  doi = {10.1145/3816146}
}
Kostenlos starten →