EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Advancing Egocentric Video Dialogue: A Contextual Reasoning Approach with New Benchmark Dataset

Jiayi Zou, Chaofan Chen

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.11.2026 · 22 (11) · 1-24

DOI: 10.1145/3847120 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 11.11.2026. Verfasst von Jiayi Zou und 1 weiteren Autor:innen. Die Arbeit nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Jiangsu Provincial Major Science and Technology Project, Postgraduate Research and Practice Innovation Program of Jiangsu Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Egocentric video reasoning capability is essential for advancing the development of first-person wearable devices. However, most existing egocentric video understanding tasks are limited to single-instance text input, overlooking the model’s capability to reason about context throughout the entire duration of the video. To bridge this gap, we propose a novel task named Egocentric Video Dialogue, which refers to answering questions by considering both the video content and dialogue history. To support this task, we design a new dataset, EgoDialogue, which constructs continuous dialogues based on video question answering datasets. Egocentric videos and dialogue histories present two challenges, requiring the model to understand core interactions in visual data and the dialogue history in text, respectively. To address these issues, we introduce an EgoReason framework, consisting of an interaction exploration module and a dialogue reasoning module. The former module first reorganizes patches that correspond to the same spatial position across different frames and then sorts these blocks according to their frame order. In this way, it models spatiotemporal dynamics to help explore interactions. The latter module improves the understanding of dialogues by gradually integrating the questions and answers from each dialogue round into the fusion layer. In the experiments, our method shows superior performance compared to several VideoQA models and vision language models, achieving state-of-the-art results in prediction accuracy and various machine translation metrics.

Förderung

  • National Natural Science Foundation of China
  • Jiangsu Provincial Major Science and Technology Project
  • Postgraduate Research and Practice Innovation Program of Jiangsu Province

Zitieren

APA

Zou, J., & Chen, C. (2026). Advancing Egocentric Video Dialogue: A Contextual Reasoning Approach with New Benchmark Dataset. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(11), 1-24. https://doi.org/10.1145/3847120

Vancouver

Zou J, Chen C. Advancing Egocentric Video Dialogue: A Contextual Reasoning Approach with New Benchmark Dataset. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(11):1-24. doi:10.1145/3847120

BibTeX
@article{Zou2026,
  author = {Zou, Jiayi and Chen, Chaofan},
  title = {Advancing Egocentric Video Dialogue: A Contextual Reasoning Approach with New Benchmark Dataset},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2026},
  volume = {22},
  number = {11},
  pages = {1--24},
  doi = {10.1145/3847120}
}
Kostenlos starten →