EUVIMEDEUVIMED entdecken
Open Access

HMM for discovering decision-making dynamics using reinforcement learning experiments

Xingche Guo, Donglin Zeng, Yuanjia Wang

Biostatistics · 2024

DOI: 10.1093/biostatistics/kxae033 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Biostatistics, 2024. Verfasst von Xingche Guo und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics, Columbia University, 722 West 168th St, New York, NY, 10032, United States. Hauptthemen laut MEDLINE: Reinforcement, Psychology, Decision Making, Major Depressive Disorder, Markov Chains. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: U.S. National Institutes of Health, NINDS NIH HHS, NIGMS NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Major depressive disorder (MDD), a leading cause of years of life lived with disability, presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD. To measure reward processing, patients perform computer-based behavioral tasks that involve making choices or responding to stimulants that are associated with different outcomes, such as gains or losses in the laboratory. Reinforcement learning (RL) models are fitted to extract parameters that measure various aspects of reward processing (e.g. reward sensitivity) to characterize how patients make decisions in behavioral tasks. Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model; instead, there may be a switching of decision-making processes between multiple strategies. An important scientific question is how the dynamics of strategies in decision-making affect the reward learning ability of individuals with MDD. Motivated by the probabilistic reward task within the Establishing Moderators and Biosignatures of Antidepressant Response in Clinical Care (EMBARC) study, we propose a novel RL-HMM (hidden Markov model) framework for analyzing reward-based decision-making. Our model accommodates decision-making strategy switching between two distinct approaches under an HMM: subjects making decisions based on the RL model or opting for random choices. We account for continuous RL state space and allow time-varying transition probabilities in the HMM. We introduce a computationally efficient Expectation-maximization (EM) algorithm for parameter estimation and use a nonparametric bootstrap for inference. Extensive simulation studies validate the finite-sample performance of our method. We apply our approach to the EMBARC study to show that MDD patients are less engaged in RL compared to the healthy controls, and engagement is associated with brain activities in the negative affect circuitry during an emotional conflict task.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Reinforcement, PsychologyDecision MakingMajor Depressive DisorderMarkov ChainsModels, StatisticalHumansRewardAdultMaleFemale

Stichwörter der Autor:innen

behavioral phenotypingbrain–behavior associationmental healthreinforcement learningreward tasksstate-switching

Autor:innen und Institutionen

  • Department of Biostatistics, Columbia University, 722 West 168th St, New York, NY, 10032, United States.
  • Department of Biostatistics, University of Michigan, 1415 Washington Heights, Ann Arbor, MI, 48109, United States.
  • Department of Psychiatry, Columbia University, 1051 Riverside Drive, New York, NY, 10032, United States.

Förderung

  • U.S. National Institutes of Health
  • NINDS NIH HHS
  • NIGMS NIH HHS
  • NHLBI NIH HHS
  • NIMH NIH HHS
  • NIH

Zitieren

APA

Guo, X., Zeng, D., & Wang, Y. (2024). HMM for discovering decision-making dynamics using reinforcement learning experiments. Biostatistics. https://doi.org/10.1093/biostatistics/kxae033

Vancouver

Guo X, Zeng D, Wang Y. HMM for discovering decision-making dynamics using reinforcement learning experiments. Biostatistics. 2024. doi:10.1093/biostatistics/kxae033

BibTeX
@article{Guo2024,
  author = {Guo, Xingche and Zeng, Donglin and Wang, Yuanjia},
  title = {HMM for discovering decision-making dynamics using reinforcement learning experiments},
  journal = {Biostatistics},
  year = {2024},
  doi = {10.1093/biostatistics/kxae033}
}
Kostenlos starten →