EUVIMEDEUVIMED entdecken
Crossref

Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges

Ying Jing, Xinyu Zhang, Mo Zhou, Yuchuan Ji, Yanchao Ding, Jialun Yin, Ruizhi Jia, Yijin Xiong, Kun Zhao, Jun Yang, Jun Li, Huaping Liu

IEEE Internet of Things Journal · 15.05.2026 · 13 (10) · 20375-20397

DOI: 10.1109/jiot.2026.3668085 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Internet of Things Journal, erschienen am 15.05.2026. Verfasst von Ying Jing und 11 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 220 Literaturangaben. Gefördert von: State Key Laboratory of Intelligent Transportation System, National Natural Science Foundation of China, Beijing Natural Science Foundation Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Autonomous vehicles rely on continuous environmental perception to assess obstacle distribution to ensure safe driving. However, current perception technologies face substantial challenges, particularly under adverse weather conditions and when encountering long-tail scenarios. With the advent of transformer-based attention mechanisms, large language models (LLMs), exemplified by GPT, have exhibited emergent intelligence, offering new possibilities for achieving high-performance perception. This technological advancement has led to the development of multimodal LLMs (MLLMs), which incorporate multimodal encoders. These models enable a single LLM to process multisource data while performing advanced understanding and reasoning tasks, enhancing complex environmental perception capabilities. Despite significant progress in MLLMs, there remains a notable gap in systematic research on their optimal application to perception tasks. Therefore, this article presents a comprehensive survey of recent advancements in MLLM-based perception. First, we introduce the mainstream vision–language perception tasks, widely adopted evaluation metrics, and existing language-enhanced autonomous driving datasets. Next, we outline the general architectural design principles of MLLMs. Subsequently, we provide a taxonomy and in-depth analysis of current MLLMs, focusing on three dimensions: input modality, alignment technique, and scene representation, elucidating their underlying implementation paradigms. Finally, we summarize the key challenges and emerging research directions in MLLM-driven perception. This survey aims to facilitate further progress in MLLMs by synthesizing these insights.

Förderung

  • State Key Laboratory of Intelligent Transportation System
  • National Natural Science Foundation of China
  • Beijing Natural Science Foundation Program

Zitieren

APA

Jing, Y., Zhang, X., Zhou, M., Ji, Y., Ding, Y., Yin, J., Jia, R., Xiong, Y., Zhao, K., Yang, J., Li, J., & Liu, H. (2026). Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges. IEEE Internet of Things Journal, 13(10), 20375-20397. https://doi.org/10.1109/jiot.2026.3668085

Vancouver

Jing Y, Zhang X, Zhou M, Ji Y, Ding Y, Yin J, et al. Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges. IEEE Internet of Things Journal. 2026;13(10):20375-20397. doi:10.1109/jiot.2026.3668085

BibTeX
@article{Jing2026,
  author = {Jing, Ying and Zhang, Xinyu and Zhou, Mo and Ji, Yuchuan and Ding, Yanchao and Yin, Jialun and Jia, Ruizhi and Xiong, Yijin and Zhao, Kun and Yang, Jun and Li, Jun and Liu, Huaping},
  title = {Multimodal Large Language Models for Perception in Autonomous Driving: Architecture, Taxonomy, and Challenges},
  journal = {IEEE Internet of Things Journal},
  year = {2026},
  volume = {13},
  number = {10},
  pages = {20375--20397},
  doi = {10.1109/jiot.2026.3668085}
}
Kostenlos starten →