EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Modality Effectiveness in Cross-Subject Sign Language Recognition: A Skeleton Perspective

Sijie Miao iD, Jiaxin He iD, Lu Zeng iD

Sensors · 16.08.2026 · 26 (16) · 5184

DOI: 10.3390/s26165184 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 16.08.2026. Verfasst von Sijie Miao und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer and Software Engineering, Xihua University, Chengdu 610039, China. Hauptthemen laut MEDLINE: Sign Language, Pattern Recognition, Automated. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Accessibility-oriented sign language recognition plays an important role in intelligent human–computer interaction, but cross-subject recognition remains challenging because test signers are unseen during training. Variations in appearance, motion style, and modality quality among different signers introduce significant distribution differences between training and testing samples, limiting model generalization. Although multimodal approaches commonly assume that RGB, skeleton, and depth modalities provide complementary information, their effectiveness under cross-subject recognition remains insufficiently explored. This study investigates modality effectiveness in SLR500 word-level sign language recognition under a cross-subject protocol. A unified evaluation framework is established to compare RGB-only, skeleton-only, RGB–skeleton fusion, and RGB–skeleton–depth fusion under consistent data splits, preprocessing procedures, training settings, and evaluation metrics. Sample-level prediction transitions are further analyzed to examine how fusion changes recognition outcomes relative to the skeleton-based reference model. The results show that the skeleton modality provides more reliable recognition performance under the evaluated cross-subject setting, while the tested fusion strategies do not consistently improve over the skeleton-only configuration. Based on the modality-effectiveness analysis, a skeleton-based reference model is established using a Spatial–Temporal Graph Convolutional Network (ST-GCN)-based spatio-temporal encoder, learned confidence-weighted temporal pooling, and a lightweight classifier. Multi-seed experiments further verify the stability of the model, which achieves 86.37% Top-1 accuracy and 96.86% Top-5 accuracy on the SLR500 cross-subject test set. These results emphasize the importance of evaluating modality reliability before designing complex fusion strategies for cross-subject sign language recognition.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Sign LanguagePattern Recognition, AutomatedHumansAlgorithmsGraph Neural Networks

Stichwörter der Autor:innen

cross-subject recognitionmodality effectivenessmultimodal fusionsign language recognitionskeleton-based recognition

Autor:innen und Institutionen

  • School of Computer and Software Engineering, Xihua University, Chengdu 610039, China.

ORCID-Profile: Sijie Miao, Jiaxin He, Lu Zeng

Zitieren

APA

Miao, S., He, J., & Zeng, L. (2026). Modality Effectiveness in Cross-Subject Sign Language Recognition: A Skeleton Perspective. Sensors, 26(16), 5184. https://doi.org/10.3390/s26165184

Vancouver

Miao S, He J, Zeng L. Modality Effectiveness in Cross-Subject Sign Language Recognition: A Skeleton Perspective. Sensors. 2026;26(16):5184. doi:10.3390/s26165184

BibTeX
@article{Miao2026,
  author = {Miao, Sijie and He, Jiaxin and Zeng, Lu},
  title = {Modality Effectiveness in Cross-Subject Sign Language Recognition: A Skeleton Perspective},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {16},
  pages = {5184},
  doi = {10.3390/s26165184}
}
Kostenlos starten →