EUVIMEDEUVIMED entdecken
Crossref

Cross-Modal Generative Semantic Communications for Mobile AIGC: Joint Semantic Encoding and Prompt Engineering

Yinqiu Liu, Hongyang Du, Dusit Niyato, Jiawen Kang, Zehui Xiong, Shiwen Mao, Ping Zhang, Xuemin Shen

IEEE Transactions on Mobile Computing · 2024-12 · 23 (12) · 14871-14888

DOI: 10.1109/tmc.2024.3449645 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Mobile Computing, 2024-12. Verfasst von Yinqiu Liu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 51 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Employing massive Mobile AI-Generated Content (AIGC) Service Providers (MASPs) with powerful models, high-quality AIGC services become accessible for resource-constrained end users. However, this advancement, referred to as mobile AIGC, also introduces a significant challenge: users should download large AIGC outputs from the MASPs, leading to substantial bandwidth consumption and potential transmission failures. In this paper, we apply cross-modal Generative Semantic Communications (G-SemCom) in mobile AIGC to overcome wireless bandwidth constraints. Specifically, we utilize cross-modal attention maps to indicate the correlation between user prompts and each part of AIGC outputs. In this way, the MASP can analyze the prompt context and filter the most semantically important content efficiently. Only semantic information is transmitted, with which users can recover the entire AIGC output with high quality while saving mobile bandwidth. Since the transmitted information not only preserves the semantics but also prompts the recovery, we formulate a joint semantic encoding and prompt engineering problem to optimize the bandwidth allocation among users. Particularly, we present a human-perceptual metric named Joint Perceptual Similarity and Quality (JPSQ), which is fused by two learning-based measurements regarding semantic similarity and aesthetic quality, respectively. Furthermore, we develop the Attention-aware Deep Diffusion (ADD) algorithm, which learns attention maps and leverages the diffusion process to enhance the environment exploration ability of traditional deep reinforcement learning (DRL). Extensive experiments demonstrate that our proposal can reduce the bandwidth consumption of mobile users by 49.4% on average, with almost no perceptual difference in AIGC output quality. Moreover, the ADD algorithm shows superior performance over baseline DRL methods, with 1.74× higher overall reward.

Zitieren

APA

Liu, Y., Du, H., Niyato, D., Kang, J., Xiong, Z., Mao, S., Zhang, P., & Shen, X. (2024). Cross-Modal Generative Semantic Communications for Mobile AIGC: Joint Semantic Encoding and Prompt Engineering. IEEE Transactions on Mobile Computing, 23(12), 14871-14888. https://doi.org/10.1109/tmc.2024.3449645

Vancouver

Liu Y, Du H, Niyato D, Kang J, Xiong Z, Mao S, et al. Cross-Modal Generative Semantic Communications for Mobile AIGC: Joint Semantic Encoding and Prompt Engineering. IEEE Transactions on Mobile Computing. 2024;23(12):14871-14888. doi:10.1109/tmc.2024.3449645

BibTeX
@article{Liu2024,
  author = {Liu, Yinqiu and Du, Hongyang and Niyato, Dusit and Kang, Jiawen and Xiong, Zehui and Mao, Shiwen and Zhang, Ping and Shen, Xuemin},
  title = {Cross-Modal Generative Semantic Communications for Mobile AIGC: Joint Semantic Encoding and Prompt Engineering},
  journal = {IEEE Transactions on Mobile Computing},
  year = {2024},
  volume = {23},
  number = {12},
  pages = {14871--14888},
  doi = {10.1109/tmc.2024.3449645}
}
Kostenlos starten →