Action- and Language-Conditioned Video Assessment for Embodied Control
Sensors · 08.08.2026 · 26 (16) · 5046
DOI: 10.3390/s26165046 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 08.08.2026. Verfasst von Hwanhee Kim und 5 weiteren Autor:innen. Beteiligte Einrichtung: Robotics Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea. Die Arbeit nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Acquisition Program Administration, Institute of Information & Communications Technology Planning & Evaluation, Center for Applied Research in Artificial Intelligence.
Abstract (Originalsprache)
Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete trajectories. Conventional approaches based on final-frame matching or continuous embedding similarity may overlook intermediate transitions that are necessary for determining whether an instruction has been completed. We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction. The method uses a pre-trained vision-language model (VLM) in two stages: it first summarizes frame-to-frame visual transitions conditioned on the executed actions and then assesses the generated summary with respect to the instruction to produce a discrete trajectory-level progress score. In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates. When used as terminal feedback for closed-loop policy optimization, it provides more effective feedback than the evaluated static image and embedding-based visual baselines and reduces the performance gap to a ground-truth oracle. These results support action- and language-conditioned video assessment as an interpretable feedback mechanism for the evaluated simulated embodied-control tasks.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Robotics Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea.
- School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea.
ORCID-Profile: Hwanhee Kim, Jaehyun Jang
Förderung
- Defense Acquisition Program Administration
- Institute of Information & Communications Technology Planning & Evaluation
- Center for Applied Research in Artificial Intelligence
- Institute for Information & communications Technology Planning & Evaluation
- National Research Foundation of Korea
Zitieren
APA
Kim, H., Jang, J., Cha, S., Yun, H., Lee, D., & Yoo, C. D. (2026). Action- and Language-Conditioned Video Assessment for Embodied Control. Sensors, 26(16), 5046. https://doi.org/10.3390/s26165046
Vancouver
Kim H, Jang J, Cha S, Yun H, Lee D, Yoo CD. Action- and Language-Conditioned Video Assessment for Embodied Control. Sensors. 2026;26(16):5046. doi:10.3390/s26165046
BibTeX
@article{Kim2026,
author = {Kim, Hwanhee and Jang, Jaehyun and Cha, Seungmin and Yun, Hyeonseo and Lee, Donghoon and Yoo, Chang D},
title = {Action- and Language-Conditioned Video Assessment for Embodied Control},
journal = {Sensors},
year = {2026},
volume = {26},
number = {16},
pages = {5046},
doi = {10.3390/s26165046}
}
EUVIMED