Vinci: A Real-time Smart Assistant Based on Egocentric Vision-language Model for Portable Devices
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 03.09.2025 · 9 (3) · 1-33
DOI: 10.1145/3749513 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 03.09.2025. Verfasst von Yifei Huang und 17 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 99 Literaturangaben.
Abstract (Originalsprache)
We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all code of Vinci is publicly available at https://github.com/OpenGVLab/vinci.
Zitieren
APA
Huang, Y., Xu, J., Pei, B., Yang, L., Zhang, M., He, Y., Chen, G., Chen, X., Wang, Y., Nie, Z., Liu, J., Lin, D., Fang, F., Li, K., Yuan, C., Qiao, Y., Wang, Y., & Wang, L. (2025). Vinci: A Real-time Smart Assistant Based on Egocentric Vision-language Model for Portable Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(3), 1-33. https://doi.org/10.1145/3749513
Vancouver
Huang Y, Xu J, Pei B, Yang L, Zhang M, He Y, et al. Vinci: A Real-time Smart Assistant Based on Egocentric Vision-language Model for Portable Devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2025;9(3):1-33. doi:10.1145/3749513
BibTeX
@article{Huang2025,
author = {Huang, Yifei and Xu, Jilan and Pei, Baoqi and Yang, Lijin and Zhang, Mingfang and He, Yuping and Chen, Guo and Chen, Xinyuan and Wang, Yaohui and Nie, Zheng and Liu, Jinyao and Lin, Dechen and Fang, Fang and Li, Kunpeng and Yuan, Chang and Qiao, Yu and Wang, Yali and Wang, Limin},
title = {Vinci: A Real-time Smart Assistant Based on Egocentric Vision-language Model for Portable Devices},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
year = {2025},
volume = {9},
number = {3},
pages = {1--33},
doi = {10.1145/3749513}
}
EUVIMED