Exploration Is a State, Not a Setting: A Markov-Switching Reinforcement-Learning Model of Strategy Transitions in Sequential Choice
Journal of Computing Theories and Applications · 21.08.2026 · 4 (1) · 387-404
DOI: 10.62411/jcta.16818 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Computing Theories and Applications, online erschienen am 21.08.2026. Verfasst von Fathimah Al-Ma'shumah und 2 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Computational accounts of human exploration usually assume a stationary policy, in which a single set of parameters for value sensitivity and uncertainty seeking generates every choice in a task, so that all within-task variation is treated as decision noise. Neuroscience instead treats exploration and exploitation as dissociable modes between which the brain switches, which a stationary model cannot represent. We introduce the Markov-Switching Reinforcement-Learning (MS-RL) model, in which a first-order hidden Markov chain governs transitions among a small number of latent decision regimes, each with its own softmax policy over a shared value-learning process. The three regimes are exploitation, directed exploration, and random exploration. The model contains the standard stationary account (one regime) and a temporally unstructured mixture (memoryless transitions) as nested special cases, making the stationarity assumption testable. We estimate the model using Expectation-Maximization with Viterbi decoding and select the number of regimes using the Bayesian information criterion. A parameter- and state-recovery study confirmed that the generating parameters and latent regime paths are recoverable (parameter correlations 0.88–0.94; state accuracy 86 percent; Cohen’s kappa ≈ 0.79). Applied to an openly available two-armed bandit dataset (46 adults, 13,800 choices), a three-regime MS-RL model was preferred over the nested baselines, two- and four-regime variants, and a single-regime model with smoothly time-varying value sensitivity. An ablation analysis attributes the largest gains to the latent regimes and temporal switching. The decoded regime path revealed a systematic within-game shift from directed exploration toward exploitation. The estimated transition matrix provides a per-participant measure of strategy change that has no counterpart in stationary models. We conclude that exploration is better described as a dynamic state than as a fixed trait, and that modeling it as a switching process is both more accurate and useful.
Zitieren
APA
Al-Ma'shumah, F., Kirani, F. F., & Ratnawaty, N. (2026). Exploration Is a State, Not a Setting: A Markov-Switching Reinforcement-Learning Model of Strategy Transitions in Sequential Choice. Journal of Computing Theories and Applications, 4(1), 387-404. https://doi.org/10.62411/jcta.16818
Vancouver
Al-Ma'shumah F, Kirani FF, Ratnawaty N. Exploration Is a State, Not a Setting: A Markov-Switching Reinforcement-Learning Model of Strategy Transitions in Sequential Choice. Journal of Computing Theories and Applications. 2026;4(1):387-404. doi:10.62411/jcta.16818
BibTeX
@article{AlMashumah2026,
author = {Al-Ma'shumah, Fathimah and Kirani, Feneta Fidi and Ratnawaty, Nita},
title = {Exploration Is a State, Not a Setting: A Markov-Switching Reinforcement-Learning Model of Strategy Transitions in Sequential Choice},
journal = {Journal of Computing Theories and Applications},
year = {2026},
volume = {4},
number = {1},
pages = {387--404},
doi = {10.62411/jcta.16818}
}
EUVIMED