EUVIMEDEUVIMED entdecken
Open Access

Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study

Shinyoung Kang iD, Michael Holcomb iD, David Hein iD, Ameer Hamza Shakur iD, Thomas Dalton iD, Andrew Jamieson iD

JMIR AI · 18.12.2025 · 4 · e76586

DOI: 10.2196/76586 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 18.12.2025. Verfasst von Shinyoung Kang und 5 weiteren Autor:innen. Beteiligte Einrichtung: Lyda Hill Department of Bioinformatics, The University of Texas Southwestern Medical Center, Dallas, TX, United States. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Objective structured clinical examinations (OSCEs) are widely used for assessing medical student competency, but their evaluation is resource-intensive, requiring trained evaluators to review 15-minute videos. The physical examination (PE) component typically constitutes only a small portion of these recordings; yet, current automated approaches struggle with processing long medical videos due to computational constraints and difficulties maintaining temporal context.

Objective

This study aims to determine whether multimodal large language models (MM-LLMs) can effectively segment PE periods within OSCE videos without previous training, potentially reducing the evaluation burden on both human graders and automated assessment systems.

Methods

We analyzed 500 videos from 5 OSCE stations at University of Texas Southwestern Simulation Center, each 15 minutes long, by using hand-labeled PE periods as ground truth. Frames were sampled at 1, 2, or 3 seconds. A pose detection preprocessing step filtered frames without people. Six MM-LLMs performed frame-level classification into encounter states by using a standardized prompt. To enforce temporal consistency, we used a hidden Markov model with Viterbi decoding, merging states into 3 primary activities (consulting/notes, physical examination, and no doctor) and adding a brief edge buffer to avoid truncating true PE segments. Performance was computed per video and averaged across the dataset by using recall, precision, intersection over union (IOU), and predicted PE length with 95% CIs.

Results

At 1-second sampling, GPT-4o achieved recall of 0.998 (95% CI 0.994-1.000), IOU of 0.784 (95% CI 0.765-0.803), and precision of 0.792 (95% CI 0.774-0.811), identifying a mean of 175 (SD 83) seconds of content per video as PE versus a mean labeled PE of 126 (SD 61) seconds, yielding an 81% reduction in video needing review (from 900 to 175 seconds). Across stations, recall remained high, with expected IOU variability linked to examination format and camera geometry. Increasing the sampling interval modestly decreased recall while slightly improving IOU and precision. Comparative baselines (eg, Gemini 2.0 Flash, Gemma 3, and Qwen2.5-VL variants) demonstrated trade-offs between recall and overselection; GPT-4o offered the best balance among high-recall models. Error analysis highlighted false negatives during occluded or verbally guided maneuvers and false positives during preparatory actions, suggesting opportunities for camera placement optimization and multimodal fusion (eg, audio cues).

Conclusions

Integrating zero-shot MM-LLMs with minimal-supervision temporal modeling effectively segments PE periods in OSCE videos without requiring extensive training data. This approach significantly reduces review time while maintaining clinical assessment integrity, demonstrating that artificial intelligence methods combining zero-shot capabilities and light supervision can be optimized for medical education’s specific requirements. This technique establishes a foundation for more efficient and scalable clinical skill assessment across diverse medical education settings.

Schlagwörter

Stichwörter der Autor:innen

AIartificial intelligencemedical educationmultimodal large language modelsvideo segmentation

Autor:innen und Institutionen

  • Lyda Hill Department of Bioinformatics, The University of Texas Southwestern Medical Center, Dallas, TX, United States.
  • Department of Internal Medicine, The University of Texas Southwestern Medical Center, Dallas, TX, United States.

ORCID-Profile: Shinyoung Kang, Michael Holcomb, David Hein, Ameer Hamza Shakur, Thomas Dalton, Andrew Jamieson

Zitieren

APA

Kang, S., Holcomb, M., Hein, D., Shakur, A. H., Dalton, T., & Jamieson, A. (2025). Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study. JMIR AI, 4, e76586. https://doi.org/10.2196/76586

Vancouver

Kang S, Holcomb M, Hein D, Shakur AH, Dalton T, Jamieson A. Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study. JMIR AI. 2025;4:e76586. doi:10.2196/76586

BibTeX
@article{Kang2025,
  author = {Kang, Shinyoung and Holcomb, Michael and Hein, David and Shakur, Ameer Hamza and Dalton, Thomas and Jamieson, Andrew},
  title = {Physical Examination Identification in Medical Education Videos: Zero-Shot Multimodal AI With Temporal Sequence Optimization Study},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e76586},
  doi = {10.2196/76586}
}
Kostenlos starten →