EUVIMEDEUVIMED entdecken
ÜbersichtsarbeitOpen Access · CC BY 4.0

Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation

Byoung Chul Ko iD

Sensors · 03.06.2026 · 26 (11) · 3541

DOI: 10.3390/s26113541 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift Sensors, online erschienen am 03.06.2026. Verfasst von Byoung Chul Ko. Beteiligte Einrichtung: Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea. Hauptthemen laut MEDLINE: Robotics. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 118 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, Ministry of Education of the Republic of Korea.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent Vision–Language–Action (VLA) models have rapidly emerged as general-purpose robotic policies that integrate language understanding, visual perception, and robot control. However, prior studies and surveys have primarily emphasized backbone architectures, action decoders, training recipes, and benchmark performance, whereas relatively limited systematic attention has been given to sensor modality selection, heterogeneous signal alignment and fusion, and their connection to action generation, all of which are critical to the performance and safety of real-world robotic manipulation. This survey addresses this gap by reinterpreting VLA within the framework of a sensor–fusion–action pipeline. This study first presents a systematic taxonomy of major sensor modalities, including RGB, depth, tactile sensing, force/torque, proprioception and inertial measurement unit, multi-spectral/thermal, and event-based vision, and compares them in terms of the physical information they provide, their characteristic failure modes, and their deployment constraints. This survey further reviews teleoperation-, human video-, and simulation-based data collection pipelines, together with representative dataset configurations, and analyzes the multi-modal design space from a sensor-centric perspective, including early and late fusion, cross-attention, token-level fusion, adapters, mixture of experts, and multi-rate action representations. In addition, this study identifies a strong bias in existing benchmarks toward RGB-centric inputs and single success-rate metrics and emphasizes the need for a multidimensional evaluation framework incorporating robustness, worst-case performance, safety, latency, and efficiency. By shifting the focus away from a model-centric narrative and explicitly accounting for real-world sensor complexity, this survey seeks to establish a sensor-centered foundation for the next generation of Physical AI.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RoboticsHumansIntelligent SystemsLanguageVisual Perception

Stichwörter der Autor:innen

Sensor–Fusion–ActionVision–Language–Actionaction representationevaluation protocolrobotic manipulationsensor fusion

Autor:innen und Institutionen

  • Department of Computer Engineering, Keimyung University, Daegu 42601, Republic of Korea.

ORCID-Profile: Byoung Chul Ko

Förderung

  • National Research Foundation of Korea
  • Ministry of Education of the Republic of Korea

Zitieren

APA

Ko, B. C. (2026). Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation. Sensors, 26(11), 3541. https://doi.org/10.3390/s26113541

Vancouver

Ko BC. Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation. Sensors. 2026;26(11):3541. doi:10.3390/s26113541

BibTeX
@article{Ko2026,
  author = {Ko, Byoung Chul},
  title = {Sensing the Action: Rethinking Sensor Modalities and Multi-Modal Fusion in Vision–Language–Action Models for Robotic Manipulation},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {11},
  pages = {3541},
  doi = {10.3390/s26113541}
}
Kostenlos starten →