EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Semantically-Enhanced Feature Extraction with CLIP and Transformer Networks for Driver Fatigue Detection

Zhen Gao, Xiaowen Chen, Jingning Xu, Rongjie Yu, Heng Zhang, Jinqiu Yang

Sensors · 12.12.2024 · 24 (24) · 7948

DOI: 10.3390/s24247948 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 12.12.2024. Verfasst von Zhen Gao und 5 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer Science and Technology, Tongji University, Shanghai 201804, China. Hauptthemen laut MEDLINE: Neural Networks, Computer, Automobile Driving, Fatigue. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the National Key R&D Program of China, the Natural Science Foundation of Shanghai, National Key R&D Program of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Drowsy driving is a leading cause of commercial vehicle traffic crashes. The trend is to train fatigue detection models using deep neural networks on driver video data, but challenges remain in coarse and incomplete high-level feature extraction and network architecture optimization. This paper pioneers the use of the CLIP (Contrastive Language-Image Pre-training) model for fatigue detection. And by harnessing the power of a Transformer architecture, sophisticated and long-term temporal features are adeptly extracted from video sequences, paving the way for more nuanced and accurate fatigue analysis. The proposed CT-Net (CLIP-Transformer Network) achieves an AUC (Area Under the Curve) of 0.892, a 36% accuracy improvement over the prevalent CNN-LSTM (Convolutional Neural Network-Long Short-Term Memory) end-to-end model, reaching state-of-the-art performance. Experiments show that the CLIP pre-trained model more accurately extracts facial and behavioral features from driver video frames, improving the model’s AUC by 7% over the ImageNet-based pre-trained model. Moreover, compared with LSTM, the Transformer more flexibly captures long-term dependencies among temporal features, further enhancing the model’s AUC by 4%.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Neural Networks, ComputerAutomobile DrivingFatigueHumansAccidents, TrafficVideo RecordingAlgorithmsSemantics

Stichwörter der Autor:innen

CLIP pre-trained modelTransformerfatigue detectioninstance normalizationsemantic analysis

Autor:innen und Institutionen

  • School of Computer Science and Technology, Tongji University, Shanghai 201804, China.
  • Department of Computer Science, City University of Hong Kong, Hong Kong 999077, China.
  • Key Laboratory of Road and Traffic Engineering of the Ministry of Education, Shanghai 201804, China.
  • College of Transportation Engineering, Tongji University, Shanghai 201804, China.
  • Zhejiang Fengxing Huiyun Technology Co., Ltd., Hangzhou 311107, China.
  • Department of Computer Science and Software Engineering, Concordia University, Montreal, QC H3G 1M8, Canada.

Förderung

  • the National Key R&D Program of China
  • the Natural Science Foundation of Shanghai
  • National Key R&D Program of China
  • Natural Science Foundation of Shanghai
  • Fundamental Research Funds for the Central Universities
  • Shanghai Municipal Science and Technology Major Project

Zitieren

APA

Gao, Z., Chen, X., Xu, J., Yu, R., Zhang, H., & Yang, J. (2024). Semantically-Enhanced Feature Extraction with CLIP and Transformer Networks for Driver Fatigue Detection. Sensors, 24(24), 7948. https://doi.org/10.3390/s24247948

Vancouver

Gao Z, Chen X, Xu J, Yu R, Zhang H, Yang J. Semantically-Enhanced Feature Extraction with CLIP and Transformer Networks for Driver Fatigue Detection. Sensors. 2024;24(24):7948. doi:10.3390/s24247948

BibTeX
@article{Gao2024,
  author = {Gao, Zhen and Chen, Xiaowen and Xu, Jingning and Yu, Rongjie and Zhang, Heng and Yang, Jinqiu},
  title = {Semantically-Enhanced Feature Extraction with CLIP and Transformer Networks for Driver Fatigue Detection},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {24},
  pages = {7948},
  doi = {10.3390/s24247948}
}
Kostenlos starten →