Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data
Engineering, Technology & Applied Science Research · 06.06.2026 · 16 (3) · 35769-35779
DOI: 10.48084/etasr.18291 ↗Kurz gefasst
Fachartikel in der Zeitschrift Engineering, Technology & Applied Science Research, online erschienen am 06.06.2026. Verfasst von Washington Kamadi und 3 weiteren Autor:innen. Die Arbeit nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Robot trajectory prediction is heavily dependent on large-scale real-world demonstrations, which limit scalability, increase data acquisition costs, and eventually prevent zero-shot generalization. To address this limitation, this paper introduces Zero-Shot Task Learning (ZSTL), a multimodal framework that uses structurally aligned synthetic data and contrastive learning to enable instruction-based trajectory generation without reliance on real-world demonstrations. ZSTL jointly encodes natural-language instructions, depth observations, LiDAR-derived spatial representations, and action trajectories within a joined embedding space, allowing cross-modal alignment and conditional behavior synthesis. The proposed architecture preserves modality structure prior to fusion by representing depth inputs as spatial tokens and LiDAR observations as temporal tokens. Together with a text token, these form a 101-token multimodal context attended over by a Transformer decoder to predict full 50-step trajectories with Gaussian uncertainty estimates. The system integrates a pretrained Bidirectional Encoder Representations from Transformers (BERT) language encoder, a ResNet-18 depth backbone, a one-dimensional convolutional LiDAR sequence encoder, and a two-layer Transformer decoder comprising approximately 125M parameters. Training was conducted entirely on a procedurally generated synthetic dataset of 5,000 samples for 50 epochs. The results demonstrate stable convergence, with the trajectory-negative log likelihood decreasing from 3.465 to -0.695 on validation data and the combined loss reaching -0.540 at epoch 18 under cosine annealed learning. The contrastive objective (InfoNCE, τ = 0.07) stabilized near 1.55, indicating consistent cross-modal alignment. The trajectory evaluation yielded an average final position error of 9.97 cm, a collision-free execution rate of 65.9%, and a task success rate of 59.2%, showing that structured synthetic supervision can support physically meaningful motion generation.
Zitieren
APA
Kamadi, W., Njiri, J. G., Kangwagye, S., & Aoki, S. (2026). Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data. Engineering, Technology & Applied Science Research, 16(3), 35769-35779. https://doi.org/10.48084/etasr.18291
Vancouver
Kamadi W, Njiri JG, Kangwagye S, Aoki S. Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data. Engineering, Technology & Applied Science Research. 2026;16(3):35769-35779. doi:10.48084/etasr.18291
BibTeX
@article{Kamadi2026,
author = {Kamadi, Washington and Njiri, Jackson Githu and Kangwagye, Samuel and Aoki, Shohei},
title = {Multimodal Contrastive Learning for Zero-Shot Instruction-Following Robot with Synthetic Data},
journal = {Engineering, Technology & Applied Science Research},
year = {2026},
volume = {16},
number = {3},
pages = {35769--35779},
doi = {10.48084/etasr.18291}
}
EUVIMED