How Good Are Large Language Models at Arithmetic Reasoning in Low-Resource Language Settings?—A Study on Yorùbá Numerical Probes with Minimal Contamination
Applied Sciences · 17.04.2025 · 15 (8) · 4459
DOI: 10.3390/app15084459 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 17.04.2025. Verfasst von Fiyinfoluwa Oyesanmi und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We study the performance of large language models (LLMs) in natural language understanding and natural language reasoning tasks in a low-resourced-language (LRL) setting. Using Yorùbá, an LRL, we curated a set of numerical probes with minimal contamination. The probes comprise three sets of questions—the first covers basic arithmetic, the second covers date and time (calendar system), and the last focuses on numerals and counting systems. Assessed in a zero-shot setup, three LLMs (ChatGPT, Gemini, and PaLM) were evaluated based on several metrics. The best-performing model, ChatGPT, generated some correct answers, showing logical steps in attaining the answers in Yorùbá (with an accuracy of 56% in set one, and 44% in set two). The second-best model (with an accuracy of 56% in set one, and 32% in set two) is Gemini. PaLM (with an accuracy of 16% in set one, and 8% in set two) showed the answers without logic. The three models performed poorly on the Yorùbá numerals question set (ChatGPT scored 8%, and Gemini and PaLM each had 0% accuracy). The study also revealed that there is significant room for improvement in the state of the art of LLMs when it comes to Yorùbá numerals.
Zitieren
APA
Oyesanmi, F., & Olukanmi, P. O. (2025). How Good Are Large Language Models at Arithmetic Reasoning in Low-Resource Language Settings?—A Study on Yorùbá Numerical Probes with Minimal Contamination. Applied Sciences, 15(8), 4459. https://doi.org/10.3390/app15084459
Vancouver
Oyesanmi F, Olukanmi PO. How Good Are Large Language Models at Arithmetic Reasoning in Low-Resource Language Settings?—A Study on Yorùbá Numerical Probes with Minimal Contamination. Applied Sciences. 2025;15(8):4459. doi:10.3390/app15084459
BibTeX
@article{Oyesanmi2025,
author = {Oyesanmi, Fiyinfoluwa and Olukanmi, Peter O.},
title = {How Good Are Large Language Models at Arithmetic Reasoning in Low-Resource Language Settings?—A Study on Yorùbá Numerical Probes with Minimal Contamination},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {8},
pages = {4459},
doi = {10.3390/app15084459}
}
EUVIMED