Benchmarking Security and Reliability of Open-Source and Commercial Large Language Models in Financial-Domain Scenarios
FinTech · 02.11.2026 · 5 (4) · 101
DOI: 10.3390/fintech5040101 ↗Kurz gefasst
Fachartikel in der Zeitschrift FinTech, online erschienen am 02.11.2026. Verfasst von Marko Pribisalić und 1 weiteren Autor:innen. Die Arbeit nennt 127 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Union.
Abstract (Originalsprache)
Large language models (LLMs) are adopted in financial services, yet existing benchmarks provide limited insight into financial-domain capability, regulatory compliance-action performance, and security robustness. This study proposes a unified multidimensional evaluation framework for evaluating open-source and commercial LLMs in regulated financial environments. The framework comprises three complementary benchmark components: the Integrated Financial Benchmark (IFB), which integrates financial-domain benchmarks for capability evaluation; the newly developed RegBench for context-grounded regulatory compliance-action performance; and the newly developed FinSec-Eval-1000 for adversarial robustness. Eight LLMs, including five open-source and three commercial models, were evaluated using an inference pipeline, task-specific metrics, statistical testing, and a Composite Security Score (SS). Claude Sonnet 4.6 achieved the highest IFB performance (68.72%) and four-judge mean security robustness (97.43%), while GPT-5.1 achieved the highest RegBench performance (50.28%). Among open-source models, Gemma 3 12B led on IFB (63.50%), whereas Phi-3-small achieved the strongest open-source RegBench (49.22%) and four-judge mean security (92.78%) results, demonstrating that model rankings vary across evaluation dimensions. No detectable monotonic association was observed between financial capability and security robustness (Spearman’s ρ = 0.095, p = 0.823). The benchmarking framework provides a transparent and reproducible methodology for multidimensional LLM evaluation and supports informed model selection in regulated financial environments.
Förderung
- European Union
Zitieren
APA
Pribisalić, M., & Martinčić-Ipšić, S. (2026). Benchmarking Security and Reliability of Open-Source and Commercial Large Language Models in Financial-Domain Scenarios. FinTech, 5(4), 101. https://doi.org/10.3390/fintech5040101
Vancouver
Pribisalić M, Martinčić-Ipšić S. Benchmarking Security and Reliability of Open-Source and Commercial Large Language Models in Financial-Domain Scenarios. FinTech. 2026;5(4):101. doi:10.3390/fintech5040101
BibTeX
@article{Pribisali2026,
author = {Pribisalić, Marko and Martinčić-Ipšić, Sanda},
title = {Benchmarking Security and Reliability of Open-Source and Commercial Large Language Models in Financial-Domain Scenarios},
journal = {FinTech},
year = {2026},
volume = {5},
number = {4},
pages = {101},
doi = {10.3390/fintech5040101}
}
EUVIMED