The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 02.12.2025 · 9 (4) · 1-32
DOI: 10.1145/3770648 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 02.12.2025. Verfasst von Ming Gao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 91 Literaturangaben. Gefördert von: the National Science Fund for Distinguished Young Scholars of China, National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province of China.
Abstract (Originalsprache)
Lip language is a straightforward method of non-verbal communication for people with laryngeal and lingual injuries, dysphonia, or vocal cord lesions. It is convenient for these speech-impaired individuals, especially when their hands are occupied. However, it depends mainly on line-of-sight-required sensors (e.g., cameras for vision-based and speaker/microphone for acoustic-based solutions) or specialized peripherals to collect and understand the lip language. Therefore, the usage of lip language is limited. Recently, inertial measurement units (IMUs) have been increasingly embedded among commercial off-the-shelf (COTS) earbuds (e.g., Apple AirPods). The built-in IMUs have the potential to sense silent articulatory gestures. However, these sensors present low sampling rates (of merely 25 Hz) and signal-to-noise ratios, blocking error-free lip reading. As a countermeasure, we comprehensively analyze the articulations of speech-impaired persons. Accordingly, we map inertial signals to lip language with effective noise cancellation. We propose an inertia-based lip language decoding system, named SwanTrumpet. It supports lip reading, unvoiced interaction, and user identification for speech-impaired users. In addition, we address the issue of user diversity in lip language translation, which is significant among the speech-impaired. We implement the prototype connected with two COTS earbuds. Extensive evaluations on 11 speech-impaired participants demonstrate its capability to improve the interaction experience for people lacking a voice.
Förderung
- the National Science Fund for Distinguished Young Scholars of China
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province of China
- the Natural Science Foundation of the Jiangsu Higher Education Institutions of China
- Natural Science Research Start-up Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications
Zitieren
APA
Gao, M., Dai, Y., Zhou, Z., Guo, Z., Han, J., & Xiao, F. (2025). The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(4), 1-32. https://doi.org/10.1145/3770648
Vancouver
Gao M, Dai Y, Zhou Z, Guo Z, Han J, Xiao F. The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2025;9(4):1-32. doi:10.1145/3770648
BibTeX
@article{Gao2025,
author = {Gao, Ming and Dai, Yichen and Zhou, Zhanbo and Guo, Zhengxin and Han, Jinsong and Xiao, Fu},
title = {The Trumpet of the Swan: Decoding Lip Language for Speech-impaired Users through COTS Earbuds},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
year = {2025},
volume = {9},
number = {4},
pages = {1--32},
doi = {10.1145/3770648}
}
EUVIMED