Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges
IEEE Internet of Things Journal · 15.05.2026 · 13 (10) · 20375-20397
DOI: 10.1109/jiot.2026.3668085 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Internet of Things Journal, erschienen am 15.05.2026. Verfasst von Ying Jing und 11 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 220 Literaturangaben. Gefördert von: State Key Laboratory of Intelligent Transportation System, National Natural Science Foundation of China, Beijing Natural Science Foundation Program.
Abstract (Originalsprache)
Autonomous vehicles rely on continuous environmental perception to assess obstacle distribution to ensure safe driving. However, current perception technologies face substantial challenges, particularly under adverse weather conditions and when encountering long-tail scenarios. With the advent of transformer-based attention mechanisms, large language models (LLMs), exemplified by GPT, have exhibited emergent intelligence, offering new possibilities for achieving high-performance perception. This technological advancement has led to the development of multimodal LLMs (MLLMs), which incorporate multimodal encoders. These models enable a single LLM to process multisource data while performing advanced understanding and reasoning tasks, enhancing complex environmental perception capabilities. Despite significant progress in MLLMs, there remains a notable gap in systematic research on their optimal application to perception tasks. Therefore, this article presents a comprehensive survey of recent advancements in MLLM-based perception. First, we introduce the mainstream vision–language perception tasks, widely adopted evaluation metrics, and existing language-enhanced autonomous driving datasets. Next, we outline the general architectural design principles of MLLMs. Subsequently, we provide a taxonomy and in-depth analysis of current MLLMs, focusing on three dimensions: input modality, alignment technique, and scene representation, elucidating their underlying implementation paradigms. Finally, we summarize the key challenges and emerging research directions in MLLM-driven perception. This survey aims to facilitate further progress in MLLMs by synthesizing these insights.
Förderung
- State Key Laboratory of Intelligent Transportation System
- National Natural Science Foundation of China
- Beijing Natural Science Foundation Program
Zitieren
APA
Jing, Y., Zhang, X., Zhou, M., Ji, Y., Ding, Y., Yin, J., Jia, R., Xiong, Y., Zhao, K., Yang, J., Li, J., & Liu, H. (2026). Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges. IEEE Internet of Things Journal, 13(10), 20375-20397. https://doi.org/10.1109/jiot.2026.3668085
Vancouver
Jing Y, Zhang X, Zhou M, Ji Y, Ding Y, Yin J, et al. Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges. IEEE Internet of Things Journal. 2026;13(10):20375-20397. doi:10.1109/jiot.2026.3668085
BibTeX
@article{Jing2026,
author = {Jing, Ying and Zhang, Xinyu and Zhou, Mo and Ji, Yuchuan and Ding, Yanchao and Yin, Jialun and Jia, Ruizhi and Xiong, Yijin and Zhao, Kun and Yang, Jun and Li, Jun and Liu, Huaping},
title = {Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges},
journal = {IEEE Internet of Things Journal},
year = {2026},
volume = {13},
number = {10},
pages = {20375--20397},
doi = {10.1109/jiot.2026.3668085}
}
EUVIMED