Personalized Federated Learning With Model-Contrastive Learning for Multi-Modal User Modeling in Human-Centric Metaverse
IEEE Journal on Selected Areas in Communications · 2024-04 · 42 (4) · 817-831
DOI: 10.1109/jsac.2023.3345431 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Journal on Selected Areas in Communications, 2024-04. Verfasst von Xiaokang Zhou und 7 weiteren Autor:innen. Die Arbeit wurde bisher 153-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Grants-in-Aid for Scientific Research (C) from Japan Society for the Promotion of Science, National Natural Science Foundation of China, Waseda University Grants for Special Research Projects.
Abstract (Originalsprache)
With the flourish of digital technologies and rapid development of 5G and beyond networks, Metaverse has become an increasingly hotly discussed topic, which offers users with multiple roles for diversified experience interacting with virtual services. How to capture and model users’ multi-platform or cross-space data/behaviors become essential to enrich people with more realistic and immersed experience in Metaverse-enabled smart applications over 5G and beyond networks. In this study, we propose a Personalized Federated Learning with Model-Contrastive Learning (PFL-MCL) framework, which may efficiently enhance the communication and interaction in human-centric Metaverse environments by making use of the large-scale, heterogeneous, and multi-modal Metaverse data. Differing from the conventional Federated Learning (FL) architecture, a multi-center aggregation structure to learn multiple global models based on the changes of dynamically updated local model weights, is developed in global, while a hierarchical neural network structure which includes a personalized module and a federated module to tackle both issues on data heterogeneity and model heterogeneity, is designed in local, so as to enhance the performance of PFL with unique characteristics of Metaverse data. In particular, a two-stage iterative clustering algorithm with a more precise initialization is developed to facilitate the personalized global aggregation with dynamically updated multiple aggregation centers. A personalized multi-modal fusion network is constructed to greatly reduce the computational cost and feature dimensions from the high-dimensional heterogeneous inputs for more efficient cross-modal fusion, based on a hierarchical shift-window attention mechanism and a newly designed bridge attention mechanism. A MCL scheme is then incorporated to speed up the model convergence with less communication overload between the local federated module and global model, while an embedding layer which effectively enables the delivered global model to better adapt to the local personality in each client is further integrated. Compared with five baseline methods, experiment and evaluation results based on two different real-world datasets demonstrate the excellent performance of our proposed PFL-MCL model in a fine-grain personalized training strategy, toward more efficient communication and networking among human-centric Metaverse enabled smart applications.
Förderung
- Grants-in-Aid for Scientific Research (C) from Japan Society for the Promotion of Science
- National Natural Science Foundation of China
- Waseda University Grants for Special Research Projects
- Shenzhen Science and Technology Program
Zitieren
APA
Zhou, X., Yang, Q., Zheng, X., Liang, W., Wang, K. I., Ma, J., Pan, Y., & Jin, Q. (2024). Personalized Federated Learning With Model-Contrastive Learning for Multi-Modal User Modeling in Human-Centric Metaverse. IEEE Journal on Selected Areas in Communications, 42(4), 817-831. https://doi.org/10.1109/jsac.2023.3345431
Vancouver
Zhou X, Yang Q, Zheng X, Liang W, Wang KI, Ma J, et al. Personalized Federated Learning With Model-Contrastive Learning for Multi-Modal User Modeling in Human-Centric Metaverse. IEEE Journal on Selected Areas in Communications. 2024;42(4):817-831. doi:10.1109/jsac.2023.3345431
BibTeX
@article{Zhou2024,
author = {Zhou, Xiaokang and Yang, Qiuyue and Zheng, Xuzhe and Liang, Wei and Wang, Kevin I-Kai and Ma, Jianhua and Pan, Yi and Jin, Qun},
title = {Personalized Federated Learning With Model-Contrastive Learning for Multi-Modal User Modeling in Human-Centric Metaverse},
journal = {IEEE Journal on Selected Areas in Communications},
year = {2024},
volume = {42},
number = {4},
pages = {817--831},
doi = {10.1109/jsac.2023.3345431}
}
EUVIMED