EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition

Neena Aloysius, Geetha M, Prema Nedungadi

IEEE Open Journal of the Computer Society · 2025 · 6 · 739-749

DOI: 10.1109/ojcs.2025.3564828 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Open Journal of the Computer Society, 2025. Verfasst von Neena Aloysius und 2 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Electronics and Information Technology, overnment of India.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study introduces Efficient ConSignformer, a novel framework advancing Continuous Sign Language Recognition (CSLR) by optimizing the Conformer-based CSLR model, ConSignformer. Central to this advancement is the Sign Query Attention (SQA) module, a computationally efficient self-attention mechanism that enhances both performance and scalability, resulting in the Efficient Conformer. Efficient ConSignformer integrates video embeddings from dual-modal CNN pipelines that process heatmaps and RGB videos, along with temporal learning layers tailored for each modality. These embeddings are further refined using the Efficient Conformer for the fused data from two modalities. To improve recognition accuracy, we employ an innovative task-adaptive supervised pretraining strategy for Efficient Conformer on a curated dataset of continuous Indian Sign Language (ISL). This strategy enables the model to effectively capture intricate data relationships during end-to-end training. Experimental results highlight the significant contributions of the SQA module and the pretraining strategy, with our model achieving competitive performance on benchmark datasets, PHOENIX-2014 and PHOENIX-2014 T. Notably, Efficient ConSignformer excels in recognizing longer sign sequences, leveraging a computationally lightweight Conformer backbone.

Förderung

  • Ministry of Electronics and Information Technology
  • overnment of India

Zitieren

APA

Aloysius, N., M, G., & Nedungadi, P. (2025). Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition. IEEE Open Journal of the Computer Society, 6, 739-749. https://doi.org/10.1109/ojcs.2025.3564828

Vancouver

Aloysius N, M G, Nedungadi P. Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition. IEEE Open Journal of the Computer Society. 2025;6:739-749. doi:10.1109/ojcs.2025.3564828

BibTeX
@article{Aloysius2025,
  author = {Aloysius, Neena and M, Geetha and Nedungadi, Prema},
  title = {Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition},
  journal = {IEEE Open Journal of the Computer Society},
  year = {2025},
  volume = {6},
  pages = {739--749},
  doi = {10.1109/ojcs.2025.3564828}
}
Kostenlos starten →