EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Action- and Language-Conditioned Video Assessment for Embodied Control

Hwanhee Kim iD, Jaehyun Jang iD, Seungmin Cha, Hyeonseo Yun, Donghoon Lee, Chang D Yoo

Sensors · 08.08.2026 · 26 (16) · 5046

DOI: 10.3390/s26165046 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 08.08.2026. Verfasst von Hwanhee Kim und 5 weiteren Autor:innen. Beteiligte Einrichtung: Robotics Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Acquisition Program Administration, Institute of Information & Communications Technology Planning & Evaluation, Center for Applied Research in Artificial Intelligence.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete trajectories. Conventional approaches based on final-frame matching or continuous embedding similarity may overlook intermediate transitions that are necessary for determining whether an instruction has been completed. We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction. The method uses a pre-trained vision-language model (VLM) in two stages: it first summarizes frame-to-frame visual transitions conditioned on the executed actions and then assesses the generated summary with respect to the instruction to produce a discrete trajectory-level progress score. In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates. When used as terminal feedback for closed-loop policy optimization, it provides more effective feedback than the evaluated static image and embedding-based visual baselines and reduces the performance gap to a ground-truth oracle. These results support action- and language-conditioned video assessment as an interpretable feedback mechanism for the evaluated simulated embodied-control tasks.

Schlagwörter

Stichwörter der Autor:innen

embodied AIembodied controltask-progress feedbacktrajectory evaluationvideo assessmentvision-language models

Autor:innen und Institutionen

  • Robotics Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea.
  • School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea.

ORCID-Profile: Hwanhee Kim, Jaehyun Jang

Förderung

  • Defense Acquisition Program Administration
  • Institute of Information & Communications Technology Planning & Evaluation
  • Center for Applied Research in Artificial Intelligence
  • Institute for Information & communications Technology Planning & Evaluation
  • National Research Foundation of Korea

Zitieren

APA

Kim, H., Jang, J., Cha, S., Yun, H., Lee, D., & Yoo, C. D. (2026). Action- and Language-Conditioned Video Assessment for Embodied Control. Sensors, 26(16), 5046. https://doi.org/10.3390/s26165046

Vancouver

Kim H, Jang J, Cha S, Yun H, Lee D, Yoo CD. Action- and Language-Conditioned Video Assessment for Embodied Control. Sensors. 2026;26(16):5046. doi:10.3390/s26165046

BibTeX
@article{Kim2026,
  author = {Kim, Hwanhee and Jang, Jaehyun and Cha, Seungmin and Yun, Hyeonseo and Lee, Donghoon and Yoo, Chang D},
  title = {Action- and Language-Conditioned Video Assessment for Embodied Control},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {16},
  pages = {5046},
  doi = {10.3390/s26165046}
}
Kostenlos starten →