Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation
Sensors · 03.06.2026 · 26 (11) · 3541
DOI: 10.3390/s26113541 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift Sensors, online erschienen am 03.06.2026. Verfasst von Byoung Chul Ko. Beteiligte Einrichtung: Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea. Hauptthemen laut MEDLINE: Robotics. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 118 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, Ministry of Education of the Republic of Korea.
Abstract (Originalsprache)
Recent Vision–Language–Action (VLA) models have rapidly emerged as general-purpose robotic policies that integrate language understanding, visual perception, and robot control. However, prior studies and surveys have primarily emphasized backbone architectures, action decoders, training recipes, and benchmark performance, whereas relatively limited systematic attention has been given to sensor modality selection, heterogeneous signal alignment and fusion, and their connection to action generation, all of which are critical to the performance and safety of real-world robotic manipulation. This survey addresses this gap by reinterpreting VLA within the framework of a sensor–fusion–action pipeline. This study first presents a systematic taxonomy of major sensor modalities, including RGB, depth, tactile sensing, force/torque, proprioception and inertial measurement unit, multi-spectral/thermal, and event-based vision, and compares them in terms of the physical information they provide, their characteristic failure modes, and their deployment constraints. This survey further reviews teleoperation-, human video-, and simulation-based data collection pipelines, together with representative dataset configurations, and analyzes the multi-modal design space from a sensor-centric perspective, including early and late fusion, cross-attention, token-level fusion, adapters, mixture of experts, and multi-rate action representations. In addition, this study identifies a strong bias in existing benchmarks toward RGB-centric inputs and single success-rate metrics and emphasizes the need for a multidimensional evaluation framework incorporating robustness, worst-case performance, safety, latency, and efficiency. By shifting the focus away from a model-centric narrative and explicitly accounting for real-world sensor complexity, this survey seeks to establish a sensor-centered foundation for the next generation of Physical AI.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.
ORCID-Profile: Byoung Chul Ko
Förderung
- National Research Foundation of Korea
- Ministry of Education of the Republic of Korea
Zitieren
APA
Ko, B. C. (2026). Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation. Sensors, 26(11), 3541. https://doi.org/10.3390/s26113541
Vancouver
Ko BC. Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation. Sensors. 2026;26(11):3541. doi:10.3390/s26113541
BibTeX
@article{Ko2026,
author = {Ko, Byoung Chul},
title = {Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation},
journal = {Sensors},
year = {2026},
volume = {26},
number = {11},
pages = {3541},
doi = {10.3390/s26113541}
}
EUVIMED