Multimodal transformers for wireless communications: A case study in beam prediction
ITU Journal on Future and Evolving Technologies · 05.09.2023 · 4 (3) · 461-471
DOI: 10.52953/jwra8095 ↗Kurz gefasst
Fachartikel in der Zeitschrift ITU Journal on Future and Evolving Technologies, erschienen am 05.09.2023. Verfasst von Yu Tian und 5 weiteren Autor:innen. Die Arbeit wurde bisher 55-mal zitiert (Crossref).
Abstract (Originalsprache)
Wireless communications at high-frequency bands with large antenna arrays face challenges in beam management, which can potentially be improved by multimodality sensing information from cameras, LiDAR, radar, and GPS. In this paper, we present a multimodal transformer deep learning framework for sensing-assisted beam prediction. We employ a convolutional neural network to extract the features from a sequence of images, point clouds, and radar raw data sampled over time. At each convolutional layer, we use transformer encoders to learn the hidden relations between feature tokens from different modalities and time instances over abstraction space and produce encoded vectors for the next-level feature extraction. We train the model on a combination of different modalities with supervised learning. We try to enhance the model over imbalanced data by utilizing focal loss and exponential moving average. We also evaluate data processing and augmentation techniques such as image enhancement, segmentation, background filtering, multimodal data flipping, radar signal transformation, and GPS angle calibration. Experimental results show that our solution trained on image and GPS data produces the best distance-based accuracy of predicted beams at 78.44%, with effective generalization to unseen day scenarios near 73% and night scenarios over 84%. This outperforms using other modalities and arbitrary data processing techniques, which demonstrates the effectiveness of transformers with feature fusion in performing radio beam prediction from images and GPS. Furthermore, our solution could be pretrained from large sequences of multimodality wireless data, on fine-tuning for multiple downstream radio network tasks.
Zitieren
APA
Tian, Y., Zhao, Q., Kherroubi, Z. E. A., Boukhalfa, F., Wu, K., & Bader, F. (2023). Multimodal transformers for wireless communications: A case study in beam prediction. ITU Journal on Future and Evolving Technologies, 4(3), 461-471. https://doi.org/10.52953/jwra8095
Vancouver
Tian Y, Zhao Q, Kherroubi ZEA, Boukhalfa F, Wu K, Bader F. Multimodal transformers for wireless communications: A case study in beam prediction. ITU Journal on Future and Evolving Technologies. 2023;4(3):461-471. doi:10.52953/jwra8095
BibTeX
@article{Tian2023,
author = {Tian, Yu and Zhao, Qiyang and Kherroubi, Zine el abidine and Boukhalfa, Fouzi and Wu, Kebin and Bader, Faouzi},
title = {Multimodal transformers for wireless communications: A case study in beam prediction},
journal = {ITU Journal on Future and Evolving Technologies},
year = {2023},
volume = {4},
number = {3},
pages = {461--471},
doi = {10.52953/jwra8095}
}
EUVIMED