Probing Multimodal LLMs as World Models for Driving
IEEE Robotics and Automation Letters · 2025-11 · 10 (11) · 11403-11410
DOI: 10.1109/lra.2025.3608656 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Robotics and Automation Letters, 2025-11. Verfasst von Shiva Sreeram und 5 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: Toyota Research Institute.
Abstract (Originalsprache)
We provide a sober look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving, challenging common assumptions about their ability to interpret dynamic driving scenarios. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored. Our experimental study assesses various MLLMs as world models using in-car camera perspectives and reveals that while these models excel at interpreting individual images, they struggle to synthesize coherent narratives across frames, leading to considerable inaccuracies in understanding (i) ego vehicle dynamics, (ii) interactions with other road actors, (iii) trajectory planning, and (iv) open-set scene reasoning. We introduce the Eval-LLM-Drive dataset and DriveSim simulator to enhance our evaluation, highlighting gaps in current MLLM capabilities and the need for improved models in dynamic real-world environments.
Förderung
- Toyota Research Institute
Zitieren
APA
Sreeram, S., Wang, T., Maalouf, A., Rosman, G., Karaman, S., & Rus, D. (2025). Probing Multimodal LLMs as World Models for Driving. IEEE Robotics and Automation Letters, 10(11), 11403-11410. https://doi.org/10.1109/lra.2025.3608656
Vancouver
Sreeram S, Wang T, Maalouf A, Rosman G, Karaman S, Rus D. Probing Multimodal LLMs as World Models for Driving. IEEE Robotics and Automation Letters. 2025;10(11):11403-11410. doi:10.1109/lra.2025.3608656
BibTeX
@article{Sreeram2025,
author = {Sreeram, Shiva and Wang, Tsun-Hsuan and Maalouf, Alaa and Rosman, Guy and Karaman, Sertac and Rus, Daniela},
title = {Probing Multimodal LLMs as World Models for Driving},
journal = {IEEE Robotics and Automation Letters},
year = {2025},
volume = {10},
number = {11},
pages = {11403--11410},
doi = {10.1109/lra.2025.3608656}
}
EUVIMED