EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Review of Reinforcement Learning With Multimodal Large Language Models for Vision-Based Human Activity Recognition

Wenqi Zheng, Yutaka Arakawa

IEEE Access · 2026 · 14 · 119987-120005

DOI: 10.1109/access.2026.3719773 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Wenqi Zheng und 1 weiteren Autor:innen. Die Arbeit nennt 117 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: JST ERATO, China Scholarship Council during the doctoral study at Kyushu University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human activity recognition (HAR) is a well-established research domain in which traditional deep learning approaches often struggle with complex human–object interactions and generalization capabilities. While multimodal large language models (MLLMs) provide strong perceptual encoding for vision-based HAR, they remain limited in handling long-horizon temporal reasoning and maintaining consistency across complex activity sequences. Reinforcement learning (RL) plays a complementary and indispensable role by enabling reward-driven optimization over extended temporal contexts, allowing MLLMs to reason over long-duration activities and adapt to dynamic HAR scenarios. This paper presents a general framework for incorporating pretrained MLLMs into HAR tasks, with a particular focus on the role of RL methods. We systematically review recent advances in the integration of MLLMs and RL for HAR, with a focus on pipeline design, policy optimization methods, reward verification, and training efficiency. Additionally, we summarize applications and existing visual datasets commonly used in HAR research. Finally, we identify key challenges and suggest future research directions to facilitate more generalizable and robust HAR systems empowered by MLLMs and RL.

Förderung

  • JST ERATO
  • China Scholarship Council during the doctoral study at Kyushu University

Zitieren

APA

Zheng, W., & Arakawa, Y. (2026). A Review of Reinforcement Learning With Multimodal Large Language Models for Vision-Based Human Activity Recognition. IEEE Access, 14, 119987-120005. https://doi.org/10.1109/access.2026.3719773

Vancouver

Zheng W, Arakawa Y. A Review of Reinforcement Learning With Multimodal Large Language Models for Vision-Based Human Activity Recognition. IEEE Access. 2026;14:119987-120005. doi:10.1109/access.2026.3719773

BibTeX
@article{Zheng2026,
  author = {Zheng, Wenqi and Arakawa, Yutaka},
  title = {A Review of Reinforcement Learning With Multimodal Large Language Models for Vision-Based Human Activity Recognition},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {119987--120005},
  doi = {10.1109/access.2026.3719773}
}
Kostenlos starten →