Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study
JMIR AI · 18.12.2025 · 4 · e76586
DOI: 10.2196/76586 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR AI, online erschienen am 18.12.2025. Verfasst von Shinyoung Kang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Lyda Hill Department of Bioinformatics, The University of Texas Southwestern Medical Center, Dallas, TX, United States. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Objective structured clinical examinations (OSCEs) are widely used for assessing medical student competency, but their evaluation is resource-intensive, requiring trained evaluators to review 15-minute videos. The physical examination (PE) component typically constitutes only a small portion of these recordings; yet, current automated approaches struggle with processing long medical videos due to computational constraints and difficulties maintaining temporal context.
Objective
This study aims to determine whether multimodal large language models (MM-LLMs) can effectively segment PE periods within OSCE videos without previous training, potentially reducing the evaluation burden on both human graders and automated assessment systems.
Methods
We analyzed 500 videos from 5 OSCE stations at University of Texas Southwestern Simulation Center, each 15 minutes long, by using hand-labeled PE periods as ground truth. Frames were sampled at 1, 2, or 3 seconds. A pose detection preprocessing step filtered frames without people. Six MM-LLMs performed frame-level classification into encounter states by using a standardized prompt. To enforce temporal consistency, we used a hidden Markov model with Viterbi decoding, merging states into 3 primary activities (consulting/notes, physical examination, and no doctor) and adding a brief edge buffer to avoid truncating true PE segments. Performance was computed per video and averaged across the dataset by using recall, precision, intersection over union (IOU), and predicted PE length with 95% CIs.
Results
At 1-second sampling, GPT-4o achieved recall of 0.998 (95% CI 0.994-1.000), IOU of 0.784 (95% CI 0.765-0.803), and precision of 0.792 (95% CI 0.774-0.811), identifying a mean of 175 (SD 83) seconds of content per video as PE versus a mean labeled PE of 126 (SD 61) seconds, yielding an 81% reduction in video needing review (from 900 to 175 seconds). Across stations, recall remained high, with expected IOU variability linked to examination format and camera geometry. Increasing the sampling interval modestly decreased recall while slightly improving IOU and precision. Comparative baselines (eg, Gemini 2.0 Flash, Gemma 3, and Qwen2.5-VL variants) demonstrated trade-offs between recall and overselection; GPT-4o offered the best balance among high-recall models. Error analysis highlighted false negatives during occluded or verbally guided maneuvers and false positives during preparatory actions, suggesting opportunities for camera placement optimization and multimodal fusion (eg, audio cues).
Conclusions
Integrating zero-shot MM-LLMs with minimal-supervision temporal modeling effectively segments PE periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that artificial intelligence methods combining zero-shot capabilities and light supervision can be optimized for medical education’s specific requirements. This technique establishes a foundation for more efficient and scalable clinical skill assessment across diverse medical education settings.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Lyda Hill Department of Bioinformatics, The University of Texas Southwestern Medical Center, Dallas, TX, United States.
- Department of Internal Medicine, The University of Texas Southwestern Medical Center, Dallas, TX, United States.
ORCID-Profile: Shinyoung Kang, Michael Holcomb, David Hein, Ameer Hamza Shakur, Thomas Dalton, Andrew Jamieson
Zitieren
APA
Kang, S., Holcomb, M., Hein, D., Shakur, A. H., Dalton, T., & Jamieson, A. (2025). Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study. JMIR AI, 4, e76586. https://doi.org/10.2196/76586
Vancouver
Kang S, Holcomb M, Hein D, Shakur AH, Dalton T, Jamieson A. Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study. JMIR AI. 2025;4:e76586. doi:10.2196/76586
BibTeX
@article{Kang2025,
author = {Kang, Shinyoung and Holcomb, Michael and Hein, David and Shakur, Ameer Hamza and Dalton, Thomas and Jamieson, Andrew},
title = {Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study},
journal = {JMIR AI},
year = {2025},
volume = {4},
pages = {e76586},
doi = {10.2196/76586}
}
EUVIMED