SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 03.03.2025 · 9 (1) · 1-30
DOI: 10.1145/3712286 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 04.03.2025. Verfasst von Bufang Yang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Science Foundation of China, Hong Kong RGC grants, National Science Foundation.
Abstract (Originalsprache)
Social interactions are fundamental to human life. The recent emergence of large language models (LLMs)-based virtual assistants has demonstrated their potential to revolutionize human interactions and lifestyles. However, existing assistive systems mainly provide reactive services to individual users, rather than offering in-situ assistance during live social interactions with conversational partners. In this study, we introduce SocialMind, the first LLM-based proactive AR social assistive system that provides users with in-situ social assistance. SocialMind employs human-like perception leveraging multi-modal sensors to extract both verbal and nonverbal cues, social factors, and implicit personas, incorporating these social cues into LLM reasoning for social suggestion generation. Additionally, SocialMind employs a multi-tier collaborative generation strategy and proactive update mechanism to display social suggestions on Augmented Reality (AR) glasses, ensuring that suggestions are timely provided to users without disrupting the natural flow of conversation. Evaluations on three public datasets and a user study with 20 participants show that SocialMind achieves 38.3% higher engagement compared to baselines, and 95% of participants are willing to use SocialMind in their live social interactions.
Förderung
- National Science Foundation of China
- Hong Kong RGC grants
- National Science Foundation
Zitieren
APA
Yang, B., Guo, Y., Xu, L., Yan, Z., Chen, H., Xing, G., & Jiang, X. (2025). SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(1), 1-30. https://doi.org/10.1145/3712286
Vancouver
Yang B, Guo Y, Xu L, Yan Z, Chen H, Xing G, et al. SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2025;9(1):1-30. doi:10.1145/3712286
BibTeX
@article{Yang2025,
author = {Yang, Bufang and Guo, Yunqi and Xu, Lilin and Yan, Zhenyu and Chen, Hongkai and Xing, Guoliang and Jiang, Xiaofan},
title = {SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
year = {2025},
volume = {9},
number = {1},
pages = {1--30},
doi = {10.1145/3712286}
}
EUVIMED