EUVIMEDObjevte EUVIMED
Mixed-Methods-StudieOpen access

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study

Alexandre Hudon iD, Barnabé Kiepura iD, Myriam Pelletier iD, Véronique Phan iD

JMIR Medical Education · 2024-04-04 · 10 · e54067-e54067

DOI: 10.2196/54067 ↗

At a glance

Mixed-Methods-Studie in JMIR Medical Education, published online on 2024-04-04. Written by Alexandre Hudon and 3 co-authors. Institution: Department of Psychiatry and Addictology, University of Montreal, Montreal, QC, Canada. Major MEDLINE topics: Education, Medical, Undergraduate, Psychiatry. Cited 35 times so far (Crossref); 38 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)–based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials.

Objective

The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion.

Methods

This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs.

Results

A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario ( P =.84), clinical questions ( P =.99), and expert opinion ( P =.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content.

Conclusions

This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

Subjects

MeSH terms (MEDLINE)

Education, Medical, UndergraduatePsychiatryHumansArtificial IntelligenceLearningCanada

Author keywords

ChatGPTaccuracyartificial intelligenceclinical questionsconcordance scriptsdesigneducationeducationaleducatorsevaluationlearningmachine learningmedical educationmedical learnerspsychiatry

Authors and institutions

  • Department of Psychiatry and Addictology, University of Montreal, Montreal, QC, Canada.
  • Faculty of Medicine, Université Laval, Québec, QC, Canada.
  • Department of Pediatrics, Université de Montréal, Montreal, QC, Canada.

ORCID profiles: Alexandre Hudon, Barnabé Kiepura, Myriam Pelletier, Véronique Phan

Cite

APA

Hudon, A., Kiepura, B., Pelletier, M., & Phan, V. (2024). Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study. JMIR Medical Education, 10, e54067-e54067. https://doi.org/10.2196/54067

Vancouver

Hudon A, Kiepura B, Pelletier M, Phan V. Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study. JMIR Medical Education. 2024;10:e54067-e54067. doi:10.2196/54067

BibTeX
@article{Hudon2024,
  author = {Hudon, Alexandre and Kiepura, Barnabé and Pelletier, Myriam and Phan, Véronique},
  title = {Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study},
  journal = {JMIR Medical Education},
  year = {2024},
  volume = {10},
  pages = {e54067--e54067},
  doi = {10.2196/54067}
}
Start for free →