Vision-language models for human motion understanding: Lessons from stroke rehabilitation
PLOS Digital Health · 06.07.2026 · 5 (7) · e0001506
DOI: 10.1371/journal.pdig.0001506 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 06.07.2026. Verfasst von Victor Li und 4 weiteren Autor:innen. Beteiligte Einrichtung: Center for Data Science, New York University, New York, New York, United States of America. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation.
Themen: Schlaganfall
Abstract (Originalsprache)
Vision–language models (VLMs) have demonstrated remarkable performance across a wide range of computer-vision tasks, sparking interest in their potential for digital health applications. Here, we apply VLMs to two fundamental challenges in data-driven stroke rehabilitation: automatic quantification of rehabilitation dose and impairment from videos. We formulate these problems as motion-identification tasks, which can be addressed using VLMs. We evaluate our proposed framework on 20 healthy controls and 51 stroke survivors. Our results show that current VLMs lack the fine-grained motion understanding required for precise quantification: dose estimates are comparable to a baseline that excludes visual information, and impairment scores cannot be reliably predicted. Nevertheless, several findings suggest future promise. With optimized prompting and post-processing, VLMs can classify high-level activities from a few frames, detect motion and grasp with moderate accuracy, and approximate dose counts within 30% of ground truth for mildly impaired and healthy participants, all without task-specific training or finetuning. These results highlight both the current limitations and emerging opportunities of VLMs for data-driven stroke rehabilitation and broader clinical video analysis.
Autor:innen und Institutionen
- Center for Data Science, New York University, New York, New York, United States of America.
- Tandon School of Engineering, New York University, Brooklyn, New York, United States of America.
- VitalConnect, San Jose, California, United States of America.
- Department of Neurology, NYU Grossman School of Medicine, New York, New York, United States of America.
- Department of Rehabilitation Medicine, NYU Grossman School of Medicine, New York, New York, United States of America.
- Courant Institute of Mathematical Sciences, New York University, New York, New York, United States of America.
ORCID-Profile: Victor Li, Heidi Schambra, Carlos Fernandez-Granda
Förderung
- National Science Foundation
Zitieren
APA
Li, V., Kamalakannan, N., Parnandi, A., Schambra, H., & Fernandez-Granda, C. (2026). Vision-language models for human motion understanding: Lessons from stroke rehabilitation. PLOS Digital Health, 5(7), e0001506. https://doi.org/10.1371/journal.pdig.0001506
Vancouver
Li V, Kamalakannan N, Parnandi A, Schambra H, Fernandez-Granda C. Vision-language models for human motion understanding: Lessons from stroke rehabilitation. PLOS Digital Health. 2026;5(7):e0001506. doi:10.1371/journal.pdig.0001506
BibTeX
@article{Li2026,
author = {Li, Victor and Kamalakannan, Naveenraj and Parnandi, Avinash and Schambra, Heidi and Fernandez-Granda, Carlos},
title = {Vision-language models for human motion understanding: Lessons from stroke rehabilitation},
journal = {PLOS Digital Health},
year = {2026},
volume = {5},
number = {7},
pages = {e0001506},
doi = {10.1371/journal.pdig.0001506}
}
EUVIMED