EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GPT-4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language models

Haicheng Liao, Huanming Shen, Zhenning Li, Chengyue Wang, Guofa Li, Yiming Bie, Chengzhong Xu

Communications in Transportation Research · 2024-12 · 4 · 100116

DOI: 10.1016/j.commtr.2023.100116 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Communications in Transportation Research, 2024-12. Verfasst von Haicheng Liao und 6 weiteren Autor:innen. Die Arbeit wurde bisher 89-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universidade de Macau, Fundo para o Desenvolvimento das Ciências e da Tecnologia.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In the field of autonomous vehicles (AVs), accurately discerning commander intent and executing linguistic commands within a visual context presents a significant challenge. This paper introduces a sophisticated encoder-decoder framework, developed to address visual grounding in AVs.Our Context-Aware Visual Grounding (CAVG) model is an advanced system that integrates five core encoders-Text, Image, Context, and Cross-Modal-with a Multimodal decoder. This integration enables the CAVG model to adeptly capture contextual semantics and to learn human emotional features, augmented by state-of-the-art Large Language Models (LLMs) including GPT-4. The architecture of CAVG is reinforced by the implementation of multi-head cross-modal attention mechanisms and a Region-Specific Dynamic (RSD) layer for attention modulation. This architectural design enables the model to efficiently process and interpret a range of cross-modal inputs, yielding a comprehensive understanding of the correlation between verbal commands and corresponding visual scenes. Empirical evaluations on the Talk2Car dataset, a real-world benchmark, demonstrate that CAVG establishes new standards in prediction accuracy and operational efficiency. Notably, the model exhibits exceptional performance even with limited training data, ranging from 50% to 75% of the full dataset. This feature highlights its effectiveness and potential for deployment in practical AV applications. Moreover, CAVG has shown remarkable robustness and adaptability in challenging scenarios, including long-text command interpretation, low-light conditions, ambiguous command contexts, inclement weather conditions, and densely populated urban environments. The code for the proposed model is available at our Github.

Förderung

  • Universidade de Macau
  • Fundo para o Desenvolvimento das Ciências e da Tecnologia

Zitieren

APA

Liao, H., Shen, H., Li, Z., Wang, C., Li, G., Bie, Y., & Xu, C. (2024). GPT-4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language models. Communications in Transportation Research, 4, 100116. https://doi.org/10.1016/j.commtr.2023.100116

Vancouver

Liao H, Shen H, Li Z, Wang C, Li G, Bie Y, et al. GPT-4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language models. Communications in Transportation Research. 2024;4:100116. doi:10.1016/j.commtr.2023.100116

BibTeX
@article{Liao2024,
  author = {Liao, Haicheng and Shen, Huanming and Li, Zhenning and Wang, Chengyue and Li, Guofa and Bie, Yiming and Xu, Chengzhong},
  title = {GPT-4 enhanced multimodal grounding for autonomous driving: Leveraging cross-modal attention with large language models},
  journal = {Communications in Transportation Research},
  year = {2024},
  volume = {4},
  pages = {100116},
  doi = {10.1016/j.commtr.2023.100116}
}
Kostenlos starten →