EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Research on the Applicability of Matrix Factorization for Ranking Large Language Models

Artyom Vyatkin, Alexander Poptsov, Valerii Oliseenko, Maxim Abramov

Informatics and Automation · 06.04.2026 · 25 (2) · 275-300

DOI: 10.15622/ia.25.2.1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Informatics and Automation, online erschienen am 06.04.2026. Verfasst von Artyom Vyatkin und 3 weiteren Autor:innen. Die Arbeit nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, Large Language Models (LLMs) have gained widespread adoption in the financial domain. Direct comparison of models can be challenging, as datasets and LLMs may be closed, and evaluation parameters may vary. This paper proposes using the matrix factorization method from recommender systems, originally designed to predict user preferences, to address the task of predicting unknown metrics. The aim is to evaluate the applicability of matrix factorization for predicting LLM performance metrics on financial tasks, as well as to develop an LLM ranking method based on metric aggregation. An experiment involving the application of matrix factorization is conducted using data collected from academic research, covering 34 LLMs and 42 financial datasets. The average Mean Absolute Error (MAE) of the method across all runs is 0.07 on the test dataset. The top positions in the ranking are held by DeepSeek-R1, OpenAI GPT-4o, OpenAI o1-mini, Fin-R1, and Claude 3.5 Sonnet. The impact of prediction error on the final results is investigated using two approaches: analysis of MAE and the Monte Carlo method. The results are analyzed, yielding the following main conclusions: a) matrix factorization can be applied to predict missing model metric values on datasets; b) leading large language models have converged in performance to such an extent that identifying a clear leader is difficult; c) large prediction errors allow for the identification of specific model features on particular tasks. The proposed method can simplify the selection of a suitable model for financial tasks.

Zitieren

APA

Vyatkin, A., Poptsov, A., Oliseenko, V., & Abramov, M. (2026). Research on the Applicability of Matrix Factorization for Ranking Large Language Models. Informatics and Automation, 25(2), 275-300. https://doi.org/10.15622/ia.25.2.1

Vancouver

Vyatkin A, Poptsov A, Oliseenko V, Abramov M. Research on the Applicability of Matrix Factorization for Ranking Large Language Models. Informatics and Automation. 2026;25(2):275-300. doi:10.15622/ia.25.2.1

BibTeX
@article{Vyatkin2026,
  author = {Vyatkin, Artyom and Poptsov, Alexander and Oliseenko, Valerii and Abramov, Maxim},
  title = {Research on the Applicability of Matrix Factorization for Ranking Large Language Models},
  journal = {Informatics and Automation},
  year = {2026},
  volume = {25},
  number = {2},
  pages = {275--300},
  doi = {10.15622/ia.25.2.1}
}
Kostenlos starten →