EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-SA 4.0

Phonesis: Towards embodied spoken language models grounded in human physiology

Mir Tahmid Hossain, Mahsa Sanaei Nourani, Zahra Rahimian, Md Nawab Yousuf Ali

Computer Science Journal of Moldova · 2026-05 · 34 (1) · 100-130

DOI: 10.56415/csjm.v34.05 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computer Science Journal of Moldova, 2026-05. Verfasst von Mir Tahmid Hossain und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Our framework Phonesis is a machine-learned model of spoken language as an embodied mechanism over real human behaviour based on multimodal corpora (MOCHA-TIMIT, GRID, VoxCeleb2). Unlike approaches based on simulations, it combines a Speaker model (for interpreting the visual input and intention and converting it into realistic speech) and a Listener model (for the audio interpretation). Phonesis is trained in the end-to-end, which means that it is highly accurate in articulatory prediction, speech quality, and intent decoding. The results of ablation studies have indicated that joint optimization enhances performance. Zero-shot evaluation VoxCeleb2 has a high degree of generalization, indicating that it can be used in voice rehabilitation, braincomputer interfaces, and language understanding

Zitieren

APA

Hossain, M. T., Nourani, M. S., Rahimian, Z., & Ali, M. N. Y. (2026). Phonesis: Towards embodied spoken language models grounded in human physiology. Computer Science Journal of Moldova, 34(1), 100-130. https://doi.org/10.56415/csjm.v34.05

Vancouver

Hossain MT, Nourani MS, Rahimian Z, Ali MNY. Phonesis: Towards embodied spoken language models grounded in human physiology. Computer Science Journal of Moldova. 2026;34(1):100-130. doi:10.56415/csjm.v34.05

BibTeX
@article{Hossain2026,
  author = {Hossain, Mir Tahmid and Nourani, Mahsa Sanaei and Rahimian, Zahra and Ali, Md Nawab Yousuf},
  title = {Phonesis: Towards embodied spoken language models grounded in human physiology},
  journal = {Computer Science Journal of Moldova},
  year = {2026},
  volume = {34},
  number = {1},
  pages = {100--130},
  doi = {10.56415/csjm.v34.05}
}
Kostenlos starten →