A Grammar-Aware Multimodal Transformer for Structured ASL-to-English Translation
Engineering, Technology & Applied Science Research · 06.06.2026 · 16 (3) · 36574-36583
DOI: 10.48084/etasr.18203 ↗Kurz gefasst
Fachartikel in der Zeitschrift Engineering, Technology & Applied Science Research, online erschienen am 06.06.2026. Verfasst von Enshirah Altarawneh und 4 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
While the process of automatically translating American Sign Language (ASL) into English remains challenging due to the inherent complexities of creating signs in space-time and due to the existence of its own grammatical structure, one of the primary objectives of this study was to create an ASL-to-English Translation Framework incorporating grammatical representations. Utilizing a formalized grammatical model of ASL rather than simply viewing ASL as a series of unconnected, unrelated signs or motions, our method views ASL as a structurally based means of communicating that is comparable to spoken languages. In addition, our system captures spatial and temporal interrelations in ASL by processing multimodal input data consisting of Red Green Blue (RGB) color video frames, 2D/3D body pose keypoints, and hand landmark information while employing a transformer-based architectural design. Moreover, we created ASL grammar tokens which represent intermediate expressions of characteristics, including whether a given sign is negative, whether a subject has been explicitly referenced, etc. The utilization of these tokens facilitates a transition from the ASL representation to the corresponding English representation. The proposed methodology was tested via experiments conducted on two publicly accessible benchmark datasets: Word-Level American Sign Language (WLASL) and Microsoft American Sign Language (MS-ASL). Results indicated that the proposed methodology outperformed the current state-of-the-art methodologies. Significant improvements in Bilingual Evaluation Understudy (BLEU-4) scores (+5.6 and +5.0 relative to baselines) were realized for WLASL and MS-ASL, respectively. Additional evaluation metrics utilized to assess increased lexical accuracy and semantic coherence included Recall-Oriented Understudy for Gisting Evaluation (ROUGE-L), Metric for Evaluation of Translation with Explicit Ordering (METEOR), and Consensus-based Image Description Evaluation (CIDEr). Lastly, our grammar token prediction module achieved an exceptionally high accuracy rate of 95.1%, thereby providing further justification for the employment of structural linguistic modeling concurrent with multimodal feature fusion within the translation pipeline. The results suggest that combining multimodal feature fusion with grammar-aware representations provides substantial improvement over previously employed methods for translating ASL into English and provides a foundation for future generations of ASL-to-English translation systems.
Zitieren
APA
Altarawneh, E., Alkasassbeh, J. S., Alshdaifat, E., Al-Qaisi, A., & Takruri, M. (2026). A Grammar-Aware Multimodal Transformer for Structured ASL-to-English Translation. Engineering, Technology & Applied Science Research, 16(3), 36574-36583. https://doi.org/10.48084/etasr.18203
Vancouver
Altarawneh E, Alkasassbeh JS, Alshdaifat E, Al-Qaisi A, Takruri M. A Grammar-Aware Multimodal Transformer for Structured ASL-to-English Translation. Engineering, Technology & Applied Science Research. 2026;16(3):36574-36583. doi:10.48084/etasr.18203
BibTeX
@article{Altarawneh2026,
author = {Altarawneh, Enshirah and Alkasassbeh, Jawdat S. and Alshdaifat, Esraa and Al-Qaisi, Aws and Takruri, Maen},
title = {A Grammar-Aware Multimodal Transformer for Structured ASL-to-English Translation},
journal = {Engineering, Technology & Applied Science Research},
year = {2026},
volume = {16},
number = {3},
pages = {36574--36583},
doi = {10.48084/etasr.18203}
}
EUVIMED