EUVIMEDEUVIMED entdecken
Open Access

A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction

Shiqiang Liu iD, Terry Fawden iD, Rong Zhu iD, George G Malliaras iD, Manohar Bance iD

Science Advances · 28.06.2024 · 10 (26)

DOI: 10.1126/sciadv.ado9576 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Science Advances, erschienen am 28.06.2024. Verfasst von Shiqiang Liu und 4 weiteren Autor:innen. Beteiligte Einrichtung: State Key Laboratory of Precision Measurement Technology and Instrument, Department of Precision Instrument, Tsinghua University, Beijing 100084, Chi…. Hauptthemen laut MEDLINE: Wearable Electronic Devices, Speech. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Lip language recognition urgently needs wearable and easy-to-use interfaces for interference-free and high-fidelity lip-reading acquisition and to develop accompanying data-efficient decoder-modeling methods. Existing solutions suffer from unreliable lip reading, are data hungry, and exhibit poor generalization. Here, we propose a wearable lip language decoding technology that enables interference-free and high-fidelity acquisition of lip movements and data-efficient recognition of fluent lip language based on wearable motion capture and continuous lip speech movement reconstruction. The method allows us to artificially generate any wanted continuous speech datasets from a very limited corpus of word samples from users. By using these artificial datasets to train the decoder, we achieve an average accuracy of 92.0% across individuals ( n = 7) for actual continuous and fluent lip speech recognition for 93 English sentences, even observing no training burn on users because all training datasets are artificially generated. Our method greatly minimizes users’ training/learning load and presents a data-efficient and easy-to-use paradigm for lip language recognition.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Wearable Electronic DevicesSpeechHumansLanguageLipMovementMaleFemaleAdultLipreadingMotion Capture

Autor:innen und Institutionen

  • State Key Laboratory of Precision Measurement Technology and Instrument, Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
  • Department of Clinical Neurosciences, University of Cambridge, Cambridge CB23EB, UK.
  • Electrical Engineering Division, Department of Engineering, University of Cambridge, Cambridge CB3 0FA, UK.

ORCID-Profile: Shiqiang Liu, Terry Fawden, Rong Zhu, George G Malliaras, Manohar Bance

Zitieren

APA

Liu, S., Fawden, T., Zhu, R., Malliaras, G. G., & Bance, M. (2024). A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction. Science Advances, 10(26). https://doi.org/10.1126/sciadv.ado9576

Vancouver

Liu S, Fawden T, Zhu R, Malliaras GG, Bance M. A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction. Science Advances. 2024;10(26). doi:10.1126/sciadv.ado9576

BibTeX
@article{Liu2024,
  author = {Liu, Shiqiang and Fawden, Terry and Zhu, Rong and Malliaras, George G and Bance, Manohar},
  title = {A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction},
  journal = {Science Advances},
  year = {2024},
  volume = {10},
  number = {26},
  doi = {10.1126/sciadv.ado9576}
}
Kostenlos starten →