EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Drivers’ Mental Engagement Analysis Using Multi-Sensor Fusion Approaches Based on Deep Convolutional Neural Networks

Taraneh Aminosharieh Najafi iD, Antonio Affanni iD, Roberto Rinaldo iD, Pamela Zontone

Sensors · 23.08.2023 · 23 (17) · 7346

DOI: 10.3390/s23177346 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 23.08.2023. Verfasst von Taraneh Aminosharieh Najafi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Polytechnic Department of Engineering and Architecture, University of Udine, Via Delle Scienze 206, 33100 Udine, Italy. Hauptthemen laut MEDLINE: Artifacts, Culture. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we present a comprehensive assessment of individuals’ mental engagement states during manual and autonomous driving scenarios using a driving simulator. Our study employed two sensor fusion approaches, combining the data and features of multimodal signals. Participants in our experiment were equipped with Electroencephalogram (EEG), Skin Potential Response (SPR), and Electrocardiogram (ECG) sensors, allowing us to collect their corresponding physiological signals. To facilitate the real-time recording and synchronization of these signals, we developed a custom-designed Graphical User Interface (GUI). The recorded signals were pre-processed to eliminate noise and artifacts. Subsequently, the cleaned data were segmented into 3 s windows and labeled according to the drivers’ high or low mental engagement states during manual and autonomous driving. To implement sensor fusion approaches, we utilized two different architectures based on deep Convolutional Neural Networks (ConvNets), specifically utilizing the Braindecode Deep4 ConvNet model. The first architecture consisted of four convolutional layers followed by a dense layer. This model processed the synchronized experimental data as a 2D array input. We also proposed a novel second architecture comprising three branches of the same ConvNet model, each with four convolutional layers, followed by a concatenation layer for integrating the ConvNet branches, and finally, two dense layers. This model received the experimental data from each sensor as a separate 2D array input for each ConvNet branch. Both architectures were evaluated using a Leave-One-Subject-Out (LOSO) cross-validation approach. For both cases, we compared the results obtained when using only EEG signals with the results obtained by adding SPR and ECG signals. In particular, the second fusion approach, using all sensor signals, achieved the highest accuracy score, reaching 82.0%. This outcome demonstrates that our proposed architecture, particularly when integrating EEG, SPR, and ECG signals at the feature level, can effectively discern the mental engagement of drivers.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

ArtifactsCultureHumansElectrocardiographyElectroencephalographyNeural Networks, Computer

Stichwörter der Autor:innen

deep convolutional neural networkdrivers’ mental engagementelectrocardiogramelectrodermal activityelectroencephalogramsensor fusion

Autor:innen und Institutionen

  • Polytechnic Department of Engineering and Architecture, University of Udine, Via Delle Scienze 206, 33100 Udine, Italy.

ORCID-Profile: Taraneh Aminosharieh Najafi, Antonio Affanni, Roberto Rinaldo

Zitieren

APA

Aminosharieh Najafi, T., Affanni, A., Rinaldo, R., & Zontone, P. (2023). Drivers’ Mental Engagement Analysis Using Multi-Sensor Fusion Approaches Based on Deep Convolutional Neural Networks. Sensors, 23(17), 7346. https://doi.org/10.3390/s23177346

Vancouver

Aminosharieh Najafi T, Affanni A, Rinaldo R, Zontone P. Drivers’ Mental Engagement Analysis Using Multi-Sensor Fusion Approaches Based on Deep Convolutional Neural Networks. Sensors. 2023;23(17):7346. doi:10.3390/s23177346

BibTeX
@article{AminoshariehNajafi2023,
  author = {Aminosharieh Najafi, Taraneh and Affanni, Antonio and Rinaldo, Roberto and Zontone, Pamela},
  title = {Drivers’ Mental Engagement Analysis Using Multi-Sensor Fusion Approaches Based on Deep Convolutional Neural Networks},
  journal = {Sensors},
  year = {2023},
  volume = {23},
  number = {17},
  pages = {7346},
  doi = {10.3390/s23177346}
}
Kostenlos starten →