EUVIMEDEUVIMED entdecken
Crossref

FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance

Yi He, Ping Wang, Shiqiang Xiong, Chao Chen, Haixiang Hu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 30987-30995

DOI: 10.1609/aaai.v40i37.40358 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yi He und 4 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Many existing financial math reasoning benchmarks suffer from data contamination and high manual construction costs. To address this, we propose a novel formula-driven approach to dynamically construct math reasoning benchmarks in finance. Our two-stage approach: (1) generates single-formula questions by LLMs using a "Mask-for-Solve" paradigm for ground truth answers, and (2) synthesizes multi-formula questions through hierarchical tree-based DAGs. Our approach ensures novelty (via LLMs' creativity) and controllability of difficulty (via DAG structure). Based on a self-constructed financial formula bank, we utilize the proposed method to build FinMathBench, the first formula-driven and fully LLM-generated benchmark aimed at assessing LLMs' math reasoning abilities in finance, containing 946 questions across 4 complexity levels. Evaluation results on 40 LLMs demonstrate significant accuracy drops in multi-formula questions, e.g., 72.9% (1-Formula) to 14.0% (4-Formula) for GPT-4o under Chain-of-Thought prompting. Three critical flaws of LLMs are also observed: poor direct calculation performance, bias toward frequently solved variables in formulas, and erroneous "correction" of valid but extreme financial values. These findings highlight gaps in current LLMs' domain-specific reasoning and underscore FinMathBench's value for advancing robust financial LLMs.

Zitieren

APA

He, Y., Wang, P., Xiong, S., Chen, C., & Hu, H. (2026). FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 30987-30995. https://doi.org/10.1609/aaai.v40i37.40358

Vancouver

He Y, Wang P, Xiong S, Chen C, Hu H. FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):30987-30995. doi:10.1609/aaai.v40i37.40358

BibTeX
@article{He2026,
  author = {He, Yi and Wang, Ping and Xiong, Shiqiang and Chen, Chao and Hu, Haixiang},
  title = {FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {37},
  pages = {30987--30995},
  doi = {10.1609/aaai.v40i37.40358}
}
Kostenlos starten →