EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters

Vilma Johnsson iD, Morten Bo Søndergaard, Kulamakan Kulasegaram, Karin Sundberg, Eleonor Tiblad, Lotta Herling, Olav Bjørn Petersen, Martin G Tolsgaard iD

Medical Education · 2024

DOI: 10.1111/medu.15190 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Medical Education, 2024. Verfasst von Vilma Johnsson und 7 weiteren Autor:innen. Beteiligte Einrichtung: Center for Fetal Medicine, Department of Obstetrics, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark. Hauptthemen laut MEDLINE: Clinical Competence, Education, Medical. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Danish Ministry of Higher Education and Science, Novo Nordisk Foundation, Novo Nordisk Fonden.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI‐based assessments (AIBA) as compared with traditional clinical expert‐based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively.

Methods

The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score.

Results

A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument.

Conclusion

There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert‐based assessments may offer complementary benefits, which is a promising subject for future research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Clinical CompetenceEducation, MedicalHumansEducational MeasurementArtificial IntelligenceReproducibility of Results

Autor:innen und Institutionen

  • Center for Fetal Medicine, Department of Obstetrics, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark.
  • Faculty of Health and Medical Science, University of Copenhagen, Copenhagen, Denmark.
  • Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark.
  • Department of Computer Science, University of Copenhagen, Copenhagen, Denmark.
  • Department of Family and Community Medicine and Scientist, Wilson Centre, Toronto, Ontario, Canada.
  • Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.

ORCID-Profile: Vilma Johnsson, Martin G Tolsgaard

Förderung

  • Danish Ministry of Higher Education and Science
  • Novo Nordisk Foundation
  • Novo Nordisk Fonden

Zitieren

APA

Johnsson, V., Søndergaard, M. B., Kulasegaram, K., Sundberg, K., Tiblad, E., Herling, L., Petersen, O. B., & Tolsgaard, M. G. (2024). Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters. Medical Education. https://doi.org/10.1111/medu.15190

Vancouver

Johnsson V, Søndergaard MB, Kulasegaram K, Sundberg K, Tiblad E, Herling L, et al. Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters. Medical Education. 2024. doi:10.1111/medu.15190

BibTeX
@article{Johnsson2024,
  author = {Johnsson, Vilma and Søndergaard, Morten Bo and Kulasegaram, Kulamakan and Sundberg, Karin and Tiblad, Eleonor and Herling, Lotta and Petersen, Olav Bjørn and Tolsgaard, Martin G},
  title = {Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters},
  journal = {Medical Education},
  year = {2024},
  doi = {10.1111/medu.15190}
}
Kostenlos starten →