EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation

Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, Lei Ma

Proceedings of the ACM on Software Engineering · 19.06.2025 · 2 (FSE) · 1615-1638

DOI: 10.1145/3729343 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 19.06.2025. Verfasst von Zhijie Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 99 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid advancement of generative Al and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model’s performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.

Zitieren

APA

Wang, Z., Zhou, Z., Song, J., Huang, Y., Shu, Z., & Ma, L. (2025). VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation. Proceedings of the ACM on Software Engineering, 2(FSE), 1615-1638. https://doi.org/10.1145/3729343

Vancouver

Wang Z, Zhou Z, Song J, Huang Y, Shu Z, Ma L. VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation. Proceedings of the ACM on Software Engineering. 2025;2(FSE):1615-1638. doi:10.1145/3729343

BibTeX
@article{Wang2025,
  author = {Wang, Zhijie and Zhou, Zhehua and Song, Jiayang and Huang, Yuheng and Shu, Zhan and Ma, Lei},
  title = {VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2025},
  volume = {2},
  number = {FSE},
  pages = {1615--1638},
  doi = {10.1145/3729343}
}
Kostenlos starten →