Deep learning-based bimodal speech and facial expression recognition of miners’ unsafe emotions
PLOS One · 15.05.2026 · 21 (5) · e0348906
DOI: 10.1371/journal.pone.0348906 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift PLOS One, online erschienen am 15.05.2026. Verfasst von Ying Lu und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Resource and Environmental Engineering, Wuhan University of Science and Technology, Wuhan, Hubei, China. Hauptthemen laut MEDLINE: Deep Learning, Miners, Safety Management, Automated Facial Recognition. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Special Project of Safety Production of the Hubei Emergency Management Department.
Abstract (Originalsprache)
Under the influence of unsafe emotions, miners’ ability to perceive risks is hindered, which can easily lead to decision-making errors and safety accidents. To recognize unsafe emotions exhibited by miners during operations, this study proposes a deep learning-based bimodal framework that integrates speech and facial expression features. A convolutional neural network (CNN) combined with a bidirectional long short-term memory (Bi-LSTM) network is employed to model local spectral patterns and temporal dependencies in speech signals, and ShuffleNet-V2 is used to capture deep facial features. In addition, three feature enhancement strategies are proposed to improve the generalization ability of the model. By constructing a dataset containing five categories of miners’ unsafe emotions for network training, the model achieves a mean recognition accuracy of 85.56%. Furthermore, we conducted a preliminary field test of the bimodal model in a real mining environment. The results provide preliminary evidence of its potential applicability in real-world mining conditions.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- School of Resource and Environmental Engineering, Wuhan University of Science and Technology, Wuhan, Hubei, China.
- Hubei Industrial Safety Engineering Technology Research Center, Wuhan, Hubei, China.
- School of Management, Fudan University, Shanghai, China.
- Hubei Chemical Safety Association, Wuhan, Hubei, China.
ORCID-Profile: Ying Lu, Zihao Zhao
Förderung
- Special Project of Safety Production of the Hubei Emergency Management Department
Zitieren
APA
Lu, Y., Zhao, Z., Yan, L., & Shi, X. (2026). Deep learning-based bimodal speech and facial expression recognition of miners’ unsafe emotions. PLOS One, 21(5), e0348906. https://doi.org/10.1371/journal.pone.0348906
Vancouver
Lu Y, Zhao Z, Yan L, Shi X. Deep learning-based bimodal speech and facial expression recognition of miners’ unsafe emotions. PLOS One. 2026;21(5):e0348906. doi:10.1371/journal.pone.0348906
BibTeX
@article{Lu2026,
author = {Lu, Ying and Zhao, Zihao and Yan, Liang and Shi, Xinyv},
title = {Deep learning-based bimodal speech and facial expression recognition of miners’ unsafe emotions},
journal = {PLOS One},
year = {2026},
volume = {21},
number = {5},
pages = {e0348906},
doi = {10.1371/journal.pone.0348906}
}
EUVIMED