EUVIMEDEUVIMED entdecken
Crossref

Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements

Weidong He, Zhi Li, Hao Wang, Tong Xu, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, Enhong Chen

ACM Transactions on Intelligent Systems and Technology · 30.06.2024 · 15 (3) · 1-25

DOI: 10.1145/3645099 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 15.04.2024. Verfasst von Weidong He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Natural Science Foundation of China, USTC Research Funds of the Double First-Class Initiative, China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The topic of multimodal conversation systems has recently garnered significant attention across various industries, including travel and retail, among others. While pioneering works in this field have shown promising performance, they often focus solely on context information at the utterance level, overlooking the context-aware dependencies of multimodal semantic elements like words and images. Furthermore, the ordinal information of images, which indicates the relevance between visual context and users’ demands, remains underutilized during the integration of visual content. Additionally, the exploration of how to effectively utilize corresponding attributes provided by users when searching for desired products is still largely unexplored. To address these challenges, we propose PMATE, a P osition-aware M ultimodal di A logue system with seman T ic E lements. Specifically, to obtain semantic representations at the element level, we first unfold the multimodal historical utterances and devise a position-aware multimodal element-level encoder. This component considers all images that may be relevant to the current turn and introduces a novel position-aware image selector to choose related images before fusing the information from the two modalities. Finally, we present a knowledge-aware two-stage decoder and an attribute-enhanced image searcher for the tasks of generating textual responses and selecting image responses, respectively. We extensively evaluate our model on two large-scale multimodal dialogue datasets, and the results of our experiments demonstrate that our approach outperforms several baseline methods.

Förderung

  • National Natural Science Foundation of China
  • USTC Research Funds of the Double First-Class Initiative
  • China Postdoctoral Science Foundation

Zitieren

APA

He, W., Li, Z., Wang, H., Xu, T., Wang, Z., Huai, B., Yuan, N. J., & Chen, E. (2024). Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-25. https://doi.org/10.1145/3645099

Vancouver

He W, Li Z, Wang H, Xu T, Wang Z, Huai B, et al. Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements. ACM Transactions on Intelligent Systems and Technology. 2024;15(3):1-25. doi:10.1145/3645099

BibTeX
@article{He2024,
  author = {He, Weidong and Li, Zhi and Wang, Hao and Xu, Tong and Wang, Zhefeng and Huai, Baoxing and Yuan, Nicholas Jing and Chen, Enhong},
  title = {Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2024},
  volume = {15},
  number = {3},
  pages = {1--25},
  doi = {10.1145/3645099}
}
Kostenlos starten →