EUVIMEDEUVIMED entdecken
Crossref

Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language

Ade Siahaan, Angelina Nadeak, Jeremy Samosir, Jesica Siburian, Boy Hutajulu

Journal of Intelligent Computing Systems and Applications · 2026

DOI: 10.54074/jicsa.v2i1.45 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Intelligent Computing Systems and Applications, 2026. Verfasst von Ade Siahaan und 4 weiteren Autor:innen. Die Arbeit nennt 26 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While Large Language Models (LLMs) demonstrate remarkable NLP capabilities, their performance on low-resource regional languages like Toba Batak remains underexplored. This study presents a rigorous empirical evaluation of four LLMs—ChatGPT, Gemini, Perplexity, and the domain-adapted Toba-LLM—on prompts written in the Toba Batak language using the CRISP-DM framework. Using a benchmark dataset spanning five task domains (information-seeking, instruction-following, creative generation, role-playing, and text-based tasks), model outputs were evaluated across six metrics: lexical similarity, keyword overlap, length ratio, multilingual cosine similarity, cultural relevance, and fluency. Experimental results reveal that general-purpose models outperform Toba-LLM in semantic similarity (cosine similarity: ChatGPT 0.628, Gemini 0.599) and fluency (ChatGPT 0.968, Gemini 0.945), whereas Toba-LLM demonstrates superior lexical alignment (0.0597) and conciseness (length ratio of 2.895). Nevertheless, cultural relevance remains a major challenge across all models (scores between 0.326 and 0.559), with qualitative findings highlighting frequent language inconsistency and Indonesian dominance bias. These findings emphasize the necessity of culturally grounded metrics, enriched regional corpora, and language-consistency constraints to advance LLM applicability in digital humanities and regional language preservation.

Zitieren

APA

Siahaan, A., Nadeak, A., Samosir, J., Siburian, J., & Hutajulu, B. (2026). Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language. Journal of Intelligent Computing Systems and Applications. https://doi.org/10.54074/jicsa.v2i1.45

Vancouver

Siahaan A, Nadeak A, Samosir J, Siburian J, Hutajulu B. Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language. Journal of Intelligent Computing Systems and Applications. 2026. doi:10.54074/jicsa.v2i1.45

BibTeX
@article{Siahaan2026,
  author = {Siahaan, Ade and Nadeak, Angelina and Samosir, Jeremy and Siburian, Jesica and Hutajulu, Boy},
  title = {Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language},
  journal = {Journal of Intelligent Computing Systems and Applications},
  year = {2026},
  doi = {10.54074/jicsa.v2i1.45}
}
Kostenlos starten →