EUVIMEDPoznaj EUVIMED
Open access · CC BY 4.0

Gaze Tracking- and Facial Movement-Driven Human–Computer Interaction System

Yue Liu, Yuxiang Li, Lu Leng, Cheonshik Kim

Applied Sciences · 2026-06-04 · 16 (11) · 5653

DOI: 10.3390/app16115653 ↗

At a glance

Journal article in Applied Sciences, published online on 2026-06-04. Written by Yue Liu and 3 co-authors. 32 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Natural Science Foundation of China, Jiangxi Provincial Key Laboratory of Image Processing and Pattern Recognition, Jiangxi Provincial Natural Science Foundation.

AbstraktDane bibliograficzneCiteRelated studies

Abstract (original language)

With the development of human–computer interaction technology, non-contact interaction based on gaze tracking and facial movements has become a research hotspot. Traditional mouse-and-keyboard methods pose challenges for people with disabilities or limited hand movements, while existing gaze-tracking systems often rely on expensive hardware or lack sufficient accuracy. This paper designs and implements a real-time system using ordinary cameras, achieving natural, efficient interaction via multimodal input combination. The system uses an improved MobileNetV2 backbone to construct GazeTrackNet for gaze estimation. It adopts MediaPipe Face Mesh to detect facial landmarks. Meanwhile, it applies geometric feature analysis, including eye aspect ratio and mouth aspect ratio, to identify actions such as blinking and mouth opening. It adopts a hybrid control strategy that combines gaze jumping and head fine-tuning, using mouth state as the main control switch. Key contributions include a lightweight gaze-tracking algorithm that enables stable and efficient gaze detection on consumer-grade hardware, a multimodal interaction strategy based on facial movement that improves system stability and ease of use, and a complete prototype system that achieves real-time performance on standard laptops. Experimental results show an average gaze average angle error of 3.0°, 97% eye state recognition accuracy, and end-to-end latency below 70 ms. The system can satisfy the requirements of daily desktop interaction under normal indoor lighting, and shows potential for future barrier-free interaction applications after further validation with target users. Existing gaze-tracking methods either suffer from low precision on lightweight devices or bring heavy computational overhead. Common facial recognition approaches also face frequent false trigger interference. Compared with them, our scheme achieves balanced accuracy and real-time performance via an attention-enhanced structure, and the designed dual anti-shake mechanism effectively suppresses misjudgment, delivering a more stable hands-free interaction experience.

Funding

  • National Natural Science Foundation of China
  • Jiangxi Provincial Key Laboratory of Image Processing and Pattern Recognition
  • Jiangxi Provincial Natural Science Foundation
  • Open Foundation of Jiangxi Provincial Key Laboratory of Image Processing and Pattern Recognition
  • Nanchang Hangkong University

Cite

APA

Liu, Y., Li, Y., Leng, L., & Kim, C. (2026). Gaze Tracking- and Facial Movement-Driven Human–Computer Interaction System. Applied Sciences, 16(11), 5653. https://doi.org/10.3390/app16115653

Vancouver

Liu Y, Li Y, Leng L, Kim C. Gaze Tracking- and Facial Movement-Driven Human–Computer Interaction System. Applied Sciences. 2026;16(11):5653. doi:10.3390/app16115653

BibTeX
@article{Liu2026,
  author = {Liu, Yue and Li, Yuxiang and Leng, Lu and Kim, Cheonshik},
  title = {Gaze Tracking- and Facial Movement-Driven Human–Computer Interaction System},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {11},
  pages = {5653},
  doi = {10.3390/app16115653}
}
Start for free →