EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication

Si Chen, Haocong Cheng, Suzy Su, Stephanie Patterson, Raja Kushalnagar, Yun Huang, Qi Wang

Proceedings of the ACM on Human-Computer Interaction · 02.05.2025 · 9 (2) · 1-31

DOI: 10.1145/3710953 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Human-Computer Interaction, online erschienen am 02.05.2025. Verfasst von Si Chen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 72 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSF.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study investigates innovative interaction designs for communication and collaborative learning between learners of mixed hearing and signing abilities, leveraging advancements in mixed reality technologies like Apple Vision Pro and generative AI for animated avatars. Adopting a participatory design approach, we engaged 15 d/Deaf and hard of hearing (DHH) students to brainstorm ideas for an AI avatar with interpreting ability (sign language to English and English to sign language) that would facilitate their face-to-face communication with hearing peers. Participants envisioned the AI avatars to address some issues with human interpreters, such as lack of availability, and provide affordable options to expensive personalized interpreting services. Our findings indicate a range of preferences for integrating the AI avatars with actual human figures of both DHH and hearing communication partners. The participants highlighted the importance of having control over customizing the AI avatar, such as AI-generated signs, voices, facial expressions, and their synchronization for enhanced emotional display in communication. Based on our findings, we propose a suite of design recommendations that balance respecting sign language norms with adherence to hearing social norms. Our study offers insights into improving the authenticity of generative AI in scenarios involving specific and sometimes unfamiliar social norms.

Förderung

  • NSF

Zitieren

APA

Chen, S., Cheng, H., Su, S., Patterson, S., Kushalnagar, R., Huang, Y., & Wang, Q. (2025). Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication. Proceedings of the ACM on Human-Computer Interaction, 9(2), 1-31. https://doi.org/10.1145/3710953

Vancouver

Chen S, Cheng H, Su S, Patterson S, Kushalnagar R, Huang Y, et al. Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication. Proceedings of the ACM on Human-Computer Interaction. 2025;9(2):1-31. doi:10.1145/3710953

BibTeX
@article{Chen2025,
  author = {Chen, Si and Cheng, Haocong and Su, Suzy and Patterson, Stephanie and Kushalnagar, Raja and Huang, Yun and Wang, Qi},
  title = {Customizing Generated Signs and Voices of AI Avatars: Deaf-Centric Mixed-Reality Design for Deaf-Hearing Communication},
  journal = {Proceedings of the ACM on Human-Computer Interaction},
  year = {2025},
  volume = {9},
  number = {2},
  pages = {1--31},
  doi = {10.1145/3710953}
}
Kostenlos starten →