EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Arithmetic Reasoning Skills Beyond Benchmark Accuracy: A Session-Based Experimental Design for LLMs

Ferhat Albayrak, Erhan Turan, Umut Orhan

Applied Sciences · 25.10.2026 · 16 (21) · 10560

DOI: 10.3390/app162110560 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 25.10.2026. Verfasst von Ferhat Albayrak und 2 weiteren Autor:innen. Die Arbeit nennt 68 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Although LLMs demonstrate high success in logical tasks, it remains debated whether this stems from genuine reasoning or statistical memorization. This study presents a novel, multi-dimensional, perturbation- and session-based behavioral testing framework to assess LLMs’ arithmetic reasoning. To reduce the potential effect of data contamination, texts from the DROP, GSM8k, and MATH datasets were structurally manipulated. A comprehensive analysis was conducted on seven different models (DeepSeek-r1, DeepSeek-v2, Gemma, Llama 3.1, Llama 3.2, Mistral, Qwen) across 90 unique base questions evaluated under 2 distinct format constraints and 4 different presentation scenarios, analyzing a total of 5040 responses. The evaluation combined three quantitative metrics with six qualitative metrics examining session-based vulnerabilities. Results reveal that, within the tested contexts, models suffered performance degradation and logical disconnections when presentation conditions, formats, or constraints changed. Furthermore, models with thinking mechanisms often failed to complete challenging tasks within the time limit rather than gaining an advantage from their reasoning process. Ultimately, these findings indicate that benchmark accuracy alone may not reflect robust reasoning, underscoring the critical need for in-depth evaluation protocols that focus on cognitive consistency and error anatomy rather than mere accuracy rates.

Zitieren

APA

Albayrak, F., Turan, E., & Orhan, U. (2026). Evaluating Arithmetic Reasoning Skills Beyond Benchmark Accuracy: A Session-Based Experimental Design for LLMs. Applied Sciences, 16(21), 10560. https://doi.org/10.3390/app162110560

Vancouver

Albayrak F, Turan E, Orhan U. Evaluating Arithmetic Reasoning Skills Beyond Benchmark Accuracy: A Session-Based Experimental Design for LLMs. Applied Sciences. 2026;16(21):10560. doi:10.3390/app162110560

BibTeX
@article{Albayrak2026,
  author = {Albayrak, Ferhat and Turan, Erhan and Orhan, Umut},
  title = {Evaluating Arithmetic Reasoning Skills Beyond Benchmark Accuracy: A Session-Based Experimental Design for LLMs},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {21},
  pages = {10560},
  doi = {10.3390/app162110560}
}
Kostenlos starten →