EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Text-to-Korean Sign Language Pose Sequence Generation Using Non-Manual Signal Conditioning and Multi-Scale Temporal Refinement

Seungju Lee iD, Gooman Park iD

Sensors · 04.07.2026 · 26 (13) · 4245

DOI: 10.3390/s26134245 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 04.07.2026. Verfasst von Seungju Lee und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Smart ICT Convergence Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republ…. Hauptthemen laut MEDLINE: Sign Language. Die Arbeit nennt 56 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & Communications Technology Planning & Evaluation, Institute of Information & Communications Technology Planning & Evaluation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automatic sign language generation has the potential to support information accessibility for deaf and hard-of-hearing individuals. Generating sign language pose sequences from natural language text can serve as an intermediate representation for avatar-based sign language expression and sign language video synthesis. However, text-to-sign pose generation is challenging because sign language conveys meaning through both manual movements and non-manual signals, while requiring temporally coherent motion over local and sentence-level contexts. In addition, text length does not directly correspond to the number of pose frames required for sign language expression. To address these issues, this study proposes a text-to-Korean Sign Language (KSL) pose generation model based on non-manual signal conditioning and multi-scale temporal refinement. The proposed framework integrates a text encoder, pose decoder, non-manual signal conditioning, multi-scale temporal refinement, and length prediction/blending. The model generates normalized 58-joint KSL keypoint sequences from morpheme-level text inputs and jointly optimizes pose reconstruction, motion continuity, bone consistency, PCK-aware precision, non-manual signal prediction, and length consistency. Experimental results on a KSL text–pose dataset show that the proposed model outperforms text-only and Transformer-based baselines. Compared with the Transformer text-to-pose baseline, the proposed model reduced MPJPE from 0.408236 to 0.316366 and Pose MAE from 0.165473 to 0.128570. It also improved PCK@0.05 from 0.136090 to 0.163928 and reduced the length relative error from 0.221455 to 0.127152. In particular, the best-threshold non-manual F1 substantially increased from 0.010859 to 0.494566. These results suggest that text-based KSL pose generation should jointly consider non-manual expressions, length consistency, and long-term temporal motion structure rather than relying only on frame-wise keypoint prediction. However, the reported improvements should be interpreted as coordinate- and label-level evidence, not as a complete validation of linguistic meaningfulness or real-world accessibility.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Sign LanguageHumansAlgorithms

Stichwörter der Autor:innen

Korean sign languageaccessibilitylength predictionmulti-scale temporal refinementnon-manual signalssign language generationtext-to-pose generation

Autor:innen und Institutionen

  • Department of Smart ICT Convergence Engineering, Seoul National University of Science and Technology, 232 Gongneung-ro, Nowon-gu, Seoul 01811, Republic of Korea.

ORCID-Profile: Seungju Lee, Gooman Park

Förderung

  • Institute of Information & Communications Technology Planning & Evaluation
  • Institute of Information & Communications Technology Planning & Evaluation

Zitieren

APA

Lee, S., & Park, G. (2026). Text-to-Korean Sign Language Pose Sequence Generation Using Non-Manual Signal Conditioning and Multi-Scale Temporal Refinement. Sensors, 26(13), 4245. https://doi.org/10.3390/s26134245

Vancouver

Lee S, Park G. Text-to-Korean Sign Language Pose Sequence Generation Using Non-Manual Signal Conditioning and Multi-Scale Temporal Refinement. Sensors. 2026;26(13):4245. doi:10.3390/s26134245

BibTeX
@article{Lee2026,
  author = {Lee, Seungju and Park, Gooman},
  title = {Text-to-Korean Sign Language Pose Sequence Generation Using Non-Manual Signal Conditioning and Multi-Scale Temporal Refinement},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {13},
  pages = {4245},
  doi = {10.3390/s26134245}
}
Kostenlos starten →