EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Creating Multimodal Interactive Digital Twin Characters From Videos: A Dataset and Baseline

Meidai Xuanyuan, Yuwang Wang, Honglei Guo, Hanshi Qu, Kun Zhang, Zhongming Li, Danping Yan, Tao Yu, Jianhua Tao, Qionghai Dai

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026

DOI: 10.1109/tpami.2025.3603653 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Meidai Xuanyuan und 9 weiteren Autor:innen. Hauptthemen laut MEDLINE: Image Processing, Computer-Assisted, User-Computer Interface, Video Recording. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 93 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we introduce a novel framework for creating multimodal interactive digital twin characters, from dialogue videos of TV shows. Specifically, these digital twin characters are capable of responding to user inputs with harmonious textual, vocal, and visual content. They not only replicate the external characteristics, such as appearance and tone, but also capture internal attributes, including personality and habitual behaviors. To support this ambitious task, we collect the Multimodal Character-Centric Conversation Dataset, named MCCCD, which includes character-specific and high-quality multimodal dialogue data with detailed annotations, featuring 6.8 k utterances and 4.6 hours of audio/video per character. Notably, the MCCCD dataset is approximately ten times larger than existing datasets in terms of per-character data volume, facilitating the detailed modeling of complex character-centric traits. Further, we propose a baseline framework to create digital twin characters, consists of dialogue generation through large language models, voice generation via speech synthesis models, and visual representation with 3D talking head models. Experimental results demonstrate that our approach significantly outperforms existing methods in generating consistent and character-specific responses, setting a new benchmark for digital character creation. Our collected dataset and proposed baseline have paved the way for the creation of highly interactive and natural digital avatars, opening the door to extensive and practical applications of digital humans.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Image Processing, Computer-AssistedUser-Computer InterfaceVideo RecordingHumansAlgorithmsDatabases, FactualTelevisionDatasets as Topic

Zitieren

APA

Xuanyuan, M., Wang, Y., Guo, H., Qu, H., Zhang, K., Li, Z., Yan, D., Yu, T., Tao, J., & Dai, Q. (2026). Creating Multimodal Interactive Digital Twin Characters From Videos: A Dataset and Baseline. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2025.3603653

Vancouver

Xuanyuan M, Wang Y, Guo H, Qu H, Zhang K, Li Z, et al. Creating Multimodal Interactive Digital Twin Characters From Videos: A Dataset and Baseline. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026. doi:10.1109/tpami.2025.3603653

BibTeX
@article{Xuanyuan2026,
  author = {Xuanyuan, Meidai and Wang, Yuwang and Guo, Honglei and Qu, Hanshi and Zhang, Kun and Li, Zhongming and Yan, Danping and Yu, Tao and Tao, Jianhua and Dai, Qionghai},
  title = {Creating Multimodal Interactive Digital Twin Characters From Videos: A Dataset and Baseline},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  doi = {10.1109/tpami.2025.3603653}
}
Kostenlos starten →