EUVIMEDObjevte EUVIMED
KohortenstudieOpen access

Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

Alice Rueda, Huda F Al-Shamali, Zack Cote, Niloy Roy iD, Reinhard Janssen-Aguilar, Jithin Joseph, Bazen Gashaw Teferra iD, Mohammad Amin Kamaleddin, Lisa Burback iD, Olga Winkler, Bill Kapralos, Andrei Torres iD and 9 more

JMIR Medical Education · 2026-07-21 · 12 · e88580-e88580

DOI: 10.2196/88580 ↗

At a glance

Kohortenstudie in JMIR Medical Education, published online on 2026-07-21. Written by Alice Rueda and 20 co-authors. Institution: Department of Psychiatry, AI for Mental Health, St. Michael's Hospital, 30 Bond Street, 17-009 Cardinal Carter South, Toronto, ON, Canada, 1416360400…. Major MEDLINE topics: Psychiatry, Referral and Consultation, Artificial Intelligence. Cited 1 times so far (Crossref); 57 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER). However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings.

Objective

This study aimed to evaluate the technical feasibility of an agentic AI system designed to support psychiatry residents’ consultation competence through simulated patient encounters with a patient agent and structured feedback from a rater agent.

Methods

We conducted a two-phase technical feasibility prospective single-arm cohort study of the STACER Agentic System, a large language model–based platform integrating a patient agent and a rater agent. Phase 1 involved automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases. Performance was assessed using DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, and role adherence) with descriptive statistics and 95% CIs. Phase 2 involved a preliminary user study with 14 convenience-sampled participants: a total of 5 members of the clinical research team and 9 psychiatry residents from the University of Alberta. Participants completed simulated diagnostic interviews and case presentations. Performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists. Interrater reliability was assessed using intraclass correlation coefficients (α=.05). Participants rated realism, behavioral consistency, psychiatric nuance, and feedback utility using Likert scales and free-text answers.

Results

The patient agent demonstrated high behavioral (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with strong automated performance (medical faithfulness mean 0.99, 95% CI 0.99‐1.00; turn relevance 0.99, 95% CI 0.986‐0.992). Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues. The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at the section subtotal level. Interrater reliability with psychiatrists was poor at the item level (intraclass correlation coefficient range=0.25‐0.49) but improved to good-to-excellent agreement at the section level for psychiatry resident sessions (intraclass correlation coefficient range=0.89‐0.93). The rater agent’s scores fell between those of the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents.

Conclusions

The STACER Agentic System demonstrates the technical feasibility of using agentic AI to simulate psychiatric consultations and deliver STACER-aligned formative feedback. By combining adaptive multiturn psychiatric simulation with competency-based evaluation, it shows promise in supporting cognitive aspects of consultation, though it remains limited in facilitating relational skills such as rapport building. These findings suggest agentic AI could expand scalable, low-risk opportunities for deliberate practice and formative feedback in competency-based psychiatric education. Further controlled studies are needed to evaluate educational effectiveness and integration into residency training.

Subjects

MeSH terms (MEDLINE)

PsychiatryReferral and ConsultationArtificial IntelligenceHumansProspective StudiesClinical CompetenceInternship and ResidencyFemaleMaleCohort StudiesAdultPsychiatristsFeasibility StudiesCanada

Author keywords

STACERagentic AI systemchatbotcompetenceconsultation skillpsychiatric residentspsychiatry educationstandardized assessment of a clinical encounter report

Authors and institutions

  • Department of Psychiatry, AI for Mental Health, St. Michael's Hospital, 30 Bond Street, 17-009 Cardinal Carter South, Toronto, ON, Canada, 14163604000, ext 76404.
  • Department of Psychiatry, University of Alberta, Edmonton, AB, Canada.
  • maxSIMhealth Group, Ontario Tech University, Oshawa, ON, Canada.
  • Department of Mathematics and Statistics, York University, Toronto, ON, Canada.
  • Department of Electrical, Computer, and Biomedical Engineering, Toronto Metropolitan University, Toronto, ON, Canada.
  • CHU Sainte-Justine Azrieli Research Center, Université de Montréal, Montréal, QC, Canada.

ORCID profiles: Niloy Roy, Bazen Gashaw Teferra, Lisa Burback, Andrei Torres, Andrew Greenshaw, Alexandre Hudon, Adam Dubrowski

Cite

APA

Rueda, A., Al-Shamali, H. F., Cote, Z., Roy, N., Janssen-Aguilar, R., Joseph, J., Teferra, B. G., Kamaleddin, M. A., Burback, L., Winkler, O., Kapralos, B., Torres, A., Sharma, D., Krishnan, S., Dumas, G., Greenshaw, A., Hudon, A., Sockalingam, S., Zhang, Y., …, & Bhat, V. (2026). Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study. JMIR Medical Education, 12, e88580-e88580. https://doi.org/10.2196/88580

Vancouver

Rueda A, Al-Shamali HF, Cote Z, Roy N, Janssen-Aguilar R, Joseph J, et al. Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study. JMIR Medical Education. 2026;12:e88580-e88580. doi:10.2196/88580

BibTeX
@article{Rueda2026,
  author = {Rueda, Alice and Al-Shamali, Huda F and Cote, Zack and Roy, Niloy and Janssen-Aguilar, Reinhard and Joseph, Jithin and Teferra, Bazen Gashaw and Kamaleddin, Mohammad Amin and Burback, Lisa and Winkler, Olga and Kapralos, Bill and Torres, Andrei and Sharma, Divya and Krishnan, Sridhar and Dumas, Guillaume and Greenshaw, Andrew and Hudon, Alexandre and Sockalingam, Sanjeev and Zhang, Yanbo and Dubrowski, Adam and Bhat, Venkat},
  title = {Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study},
  journal = {JMIR Medical Education},
  year = {2026},
  volume = {12},
  pages = {e88580--e88580},
  doi = {10.2196/88580}
}
Start for free →