A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints
Information Technology and Computer Engineering · 03.08.2026 · (23) · 55-66
DOI: 10.31649/vitce/2.2026.55 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information Technology and Computer Engineering, online erschienen am 03.08.2026. Verfasst von Artur Markov. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The study aimed to justify an approach to the planning and completion of Bayesian A/B tests, within which interval certainty criteria are combined with threshold requirements for a practically significant effect and an assessment of the economic feasibility of further data collection. The methodology was based on methods of analytical synthesis, structural-logical classification, logical-algorithmic formalisation, and illustrative case analysis. The study established that in Bayesian A/B tests, “interval precision” has two dimensions: informativeness (the narrowness and stability of the credible interval for supporting an engineering decision) and calibratedness (the proximity of actual coverage to the nominal level, which may deviate for small samples). The study demonstrated that uncertainty decreases with increasing n but with diminishing returns; therefore, further narrowing of intervals requires disproportionately greater expenditure of time and computational or experimental resources and is justified only when such costs are warranted, with the required n depending on the metric and the scale of the effect. The volume of data can be reduced by incorporating prior information into the prior distribution or by defining “sufficiency” based on criteria of evidence or the probability of achieving the objective, whereas high variability of engineering metrics, rare events, low baseline event rates, strict accuracy requirements, and a small effect increase the required n and the duration of the test. To reduce the required volume of data, a sequential framework for concluding a Bayesian A/B test has been proposed, combining an information component, a target component, and an economic feasibility check: “update estimates → check threshold → check interval → weigh up the benefits and costs of continuing”. The practical applicability of this approach was confirmed by engineering studies, where Bayesian methods reduced resource consumption by 66.6%, 36.7%, and 26.9% in stream computing and achieved up to 25% energy savings, 15% greater energy efficiency, and about 10 ms tighter latency guarantees in edge computing. The practical significance of the results lies in their applicability to researchers and analysts involved in engineering experiments when making decisions regarding the termination of the experiment and the implementation of the proposed changes
Zitieren
APA
Markov, A. (2026). A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints. Information Technology and Computer Engineering, 55-66. https://doi.org/10.31649/vitce/2.2026.55
Vancouver
Markov A. A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints. Information Technology and Computer Engineering. 2026:55-66. doi:10.31649/vitce/2.2026.55
BibTeX
@article{Markov2026,
author = {Markov, Artur},
title = {A sequential decision framework for Bayesian A/B Tests: Reconciling interval precision, practical significance, and resource constraints},
journal = {Information Technology and Computer Engineering},
year = {2026},
number = {23},
pages = {55--66},
doi = {10.31649/vitce/2.2026.55}
}
EUVIMED