Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements
ACM Transactions on Intelligent Systems and Technology · 30.06.2024 · 15 (3) · 1-25
DOI: 10.1145/3645099 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 15.04.2024. Verfasst von Weidong He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Natural Science Foundation of China, USTC Research Funds of the Double First-Class Initiative, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
The topic of multimodal conversation systems has recently garnered significant attention across various industries, including travel and retail, among others. While pioneering works in this field have shown promising performance, they often focus solely on context information at the utterance level, overlooking the context-aware dependencies of multimodal semantic elements like words and images. Furthermore, the ordinal information of images, which indicates the relevance between visual context and users’ demands, remains underutilized during the integration of visual content. Additionally, the exploration of how to effectively utilize corresponding attributes provided by users when searching for desired products is still largely unexplored. To address these challenges, we propose PMATE, a P osition-aware M ultimodal di A logue system with seman T ic E lements. Specifically, to obtain semantic representations at the element level, we first unfold the multimodal historical utterances and devise a position-aware multimodal element-level encoder. This component considers all images that may be relevant to the current turn and introduces a novel position-aware image selector to choose related images before fusing the information from the two modalities. Finally, we present a knowledge-aware two-stage decoder and an attribute-enhanced image searcher for the tasks of generating textual responses and selecting image responses, respectively. We extensively evaluate our model on two large-scale multimodal dialogue datasets, and the results of our experiments demonstrate that our approach outperforms several baseline methods.
Förderung
- National Natural Science Foundation of China
- USTC Research Funds of the Double First-Class Initiative
- China Postdoctoral Science Foundation
Zitieren
APA
He, W., Li, Z., Wang, H., Xu, T., Wang, Z., Huai, B., Yuan, N. J., & Chen, E. (2024). Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements. ACM Transactions on Intelligent Systems and Technology, 15(3), 1-25. https://doi.org/10.1145/3645099
Vancouver
He W, Li Z, Wang H, Xu T, Wang Z, Huai B, et al. Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements. ACM Transactions on Intelligent Systems and Technology. 2024;15(3):1-25. doi:10.1145/3645099
BibTeX
@article{He2024,
author = {He, Weidong and Li, Zhi and Wang, Hao and Xu, Tong and Wang, Zhefeng and Huai, Baoxing and Yuan, Nicholas Jing and Chen, Enhong},
title = {Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2024},
volume = {15},
number = {3},
pages = {1--25},
doi = {10.1145/3645099}
}
EUVIMED