EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Characterizing LLM performance via a Bayesian lens

Giannis Manousaridis, John G. Samuelsson, Birol Emir, Shibasish Dasgupta, Gaurav Jain, Subha Madhavan, Sheraz Khan

Frontiers in Applied Mathematics and Statistics · 31.07.2026 · 12

DOI: 10.3389/fams.2026.1854495 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Applied Mathematics and Statistics, online erschienen am 31.07.2026. Verfasst von Giannis Manousaridis und 6 weiteren Autor:innen. Die Arbeit nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While large language models (LLMs) are having a transformative impact on human society, evaluating them remains challenging. Standard benchmarks usually rely on single point estimates that obscure response stochasticity and variability in question difficulty. Here, we introduce a hierarchical Bayesian Beta-Binomial framework for uncertainty-aware evaluation of LLMs on multiple-choice datasets. Our approach moves beyond single accuracy metrics by modeling correct responses binomially and decomposing performance variation into intra-question stochasticity (response variability for a given question) and inter-question heterogeneity (variation in difficulty across questions) using separate priors. The framework yields a probabilistic assessment, providing full posterior distributions and credible intervals for mean accuracy, inter-question heterogeneity, and mean intra-question response variability, enabling rigorous uncertainty quantification. We demonstrate its utility by evaluating multiple LLMs across diverse benchmarks, including under semantic perturbations like question rephrasing. This analysis reveals nuanced model robustness insights and uncovers distinct behaviors across model classes (e.g., reasoning vs. non-reasoning) often missed by traditional descriptive statistics. This methodology offers a statistically grounded and powerful Bayesian lens for analyzing LLM performance, providing deeper insights into accuracy, consistency, and heterogeneity essential for reliable model benchmarking.

Zitieren

APA

Manousaridis, G., Samuelsson, J. G., Emir, B., Dasgupta, S., Jain, G., Madhavan, S., & Khan, S. (2026). Characterizing LLM performance via a Bayesian lens. Frontiers in Applied Mathematics and Statistics, 12. https://doi.org/10.3389/fams.2026.1854495

Vancouver

Manousaridis G, Samuelsson JG, Emir B, Dasgupta S, Jain G, Madhavan S, et al. Characterizing LLM performance via a Bayesian lens. Frontiers in Applied Mathematics and Statistics. 2026;12. doi:10.3389/fams.2026.1854495

BibTeX
@article{Manousaridis2026,
  author = {Manousaridis, Giannis and Samuelsson, John G. and Emir, Birol and Dasgupta, Shibasish and Jain, Gaurav and Madhavan, Subha and Khan, Sheraz},
  title = {Characterizing LLM performance via a Bayesian lens},
  journal = {Frontiers in Applied Mathematics and Statistics},
  year = {2026},
  volume = {12},
  doi = {10.3389/fams.2026.1854495}
}
Kostenlos starten →