Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions
Sensors · 27.05.2024 · 24 (11) · 3440
DOI: 10.3390/s24113440 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 27.05.2024. Verfasst von Yuya Moroto und 4 weiteren Autor:innen. Beteiligte Einrichtung: Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science, JSPS KAKENHI.
Abstract (Originalsprache)
This paper proposes a multimodal Transformer model that uses time-series data to detect and predict winter road surface conditions. For detecting or predicting road surface conditions, the previous approach focuses on the cooperative use of multiple modalities as inputs, e.g., images captured by fixed-point cameras (road surface images) and auxiliary data related to road surface conditions under simple modality integration. Although such an approach achieves performance improvement compared to the method using only images or auxiliary data, there is a demand for further consideration of the way to integrate heterogeneous modalities. The proposed method realizes a more effective modality integration using a cross-attention mechanism and time-series processing. Concretely, when integrating multiple modalities, feature compensation through mutual complementation between modalities is realized through a feature integration technique based on a cross-attention mechanism, and the representational ability of the integrated features is enhanced. In addition, by introducing time-series processing for the input data across several timesteps, it is possible to consider the temporal changes in the road surface conditions. Experiments are conducted for both detection and prediction tasks using data corresponding to the current winter condition and data corresponding to a few hours after the current winter condition, respectively. The experimental results verify the effectiveness of the proposed method for both tasks. In addition to the construction of the classification model for winter road surface conditions, we first attempt to visualize the classification results, especially the prediction results, through the image style transfer model as supplemental extended experiments on image generation at the end of the paper.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan.
- Data-Driven Interdisciplinary Research Emergence Department, Hokkaido University, N-13, W-10, Kita-ku, Sapporo 060-0813, Japan.
- Faculty of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo 060-0814, Japan.
ORCID-Profile: Yuya Moroto, Keisuke Maeda, Ren Togo, Takahiro Ogawa, Miki Haseyama
Förderung
- Japan Society for the Promotion of Science
- JSPS KAKENHI
Zitieren
APA
Moroto, Y., Maeda, K., Togo, R., Ogawa, T., & Haseyama, M. (2024). Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions. Sensors, 24(11), 3440. https://doi.org/10.3390/s24113440
Vancouver
Moroto Y, Maeda K, Togo R, Ogawa T, Haseyama M. Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions. Sensors. 2024;24(11):3440. doi:10.3390/s24113440
BibTeX
@article{Moroto2024,
author = {Moroto, Yuya and Maeda, Keisuke and Togo, Ren and Ogawa, Takahiro and Haseyama, Miki},
title = {Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions},
journal = {Sensors},
year = {2024},
volume = {24},
number = {11},
pages = {3440},
doi = {10.3390/s24113440}
}
EUVIMED