FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 30987-30995
DOI: 10.1609/aaai.v40i37.40358 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yi He und 4 weiteren Autor:innen.
Abstract (Originalsprache)
Many existing financial math reasoning benchmarks suffer from data contamination and high manual construction costs. To address this, we propose a novel formula-driven approach to dynamically construct math reasoning benchmarks in finance. Our two-stage approach: (1) generates single-formula questions by LLMs using a "Mask-for-Solve" paradigm for ground truth answers, and (2) synthesizes multi-formula questions through hierarchical tree-based DAGs. Our approach ensures novelty (via LLMs' creativity) and controllability of difficulty (via DAG structure). Based on a self-constructed financial formula bank, we utilize the proposed method to build FinMathBench, the first formula-driven and fully LLM-generated benchmark aimed at assessing LLMs' math reasoning abilities in finance, containing 946 questions across 4 complexity levels. Evaluation results on 40 LLMs demonstrate significant accuracy drops in multi-formula questions, e.g., 72.9% (1-Formula) to 14.0% (4-Formula) for GPT-4o under Chain-of-Thought prompting. Three critical flaws of LLMs are also observed: poor direct calculation performance, bias toward frequently solved variables in formulas, and erroneous "correction" of valid but extreme financial values. These findings highlight gaps in current LLMs' domain-specific reasoning and underscore FinMathBench's value for advancing robust financial LLMs.
Zitieren
APA
He, Y., Wang, P., Xiong, S., Chen, C., & Hu, H. (2026). FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 30987-30995. https://doi.org/10.1609/aaai.v40i37.40358
Vancouver
He Y, Wang P, Xiong S, Chen C, Hu H. FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):30987-30995. doi:10.1609/aaai.v40i37.40358
BibTeX
@article{He2026,
author = {He, Yi and Wang, Ping and Xiong, Shiqiang and Chen, Chao and Hu, Haixiang},
title = {FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {37},
pages = {30987--30995},
doi = {10.1609/aaai.v40i37.40358}
}
EUVIMED