F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (17) · 14493-14501
DOI: 10.1609/aaai.v40i17.38466 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Hanbo Bi und 8 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Traditional dialogue retrieval aims to select the most appropriate utterance or image from recent dialogue history. However, they often fail to meet users’ actual needs for revisiting semantically coherent content scattered across long-form conversations. To fill this gap, we define the Fine-grained Fragment Retrieval (FFR) task, requiring models to locate query-relevant fragments, comprising both utterances and images, from multimodal long-form dialogues. As a foundation for FFR, we construct MLDR, the longest-turn multimodal dialogue retrieval dataset to date, averaging 25.45 turns per dialogue, with each naturally spanning three distinct topics. To evaluate generalization in real-world scenarios, we curate and annotate a WeChat-based test set comprising real-world multimodal dialogues with an average of 75.38 turns. Building on these resources, we explore existing generation-based Vision-Language Models (VLMs) on FFR and observe that they often retrieve incoherent utterance-image fragments. While optimized for generating responses from visual-textual inputs, these models lack explicit supervision to ensure semantic coherence within retrieved fragments. To address this, we propose F2RVLM, a generative retrieval model trained in a two-stage paradigm: (1) supervised fine-tuning to inject fragment-level retrieval knowledge, and (2) GRPO-based reinforcement learning with multi-objective rewards to encourage outputs with semantic precision, relevance, and contextual coherence. In addition, to account for difficulty variations arising from differences in intra-fragment element distribution, ranging from locally dense to sparsely scattered, we introduce a difficulty-aware curriculum sampling that ranks training instances by predicted difficulty and gradually incorporates harder examples. This strategy enhances the model’s reasoning ability in long-form, multi-turn dialogue contexts. Experiments on both in-domain and real-domain sets demonstrate that F2RVLM substantially outperforms popular VLMs, achieving superior retrieval performance.
Zitieren
APA
Bi, H., Yuan, Z., Jia, Z., Zhang, J., Li, C., Luo, P., Deng, Y., Duan, X., & Zhang, J. (2026). F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model. Proceedings of the AAAI Conference on Artificial Intelligence, 40(17), 14493-14501. https://doi.org/10.1609/aaai.v40i17.38466
Vancouver
Bi H, Yuan Z, Jia Z, Zhang J, Li C, Luo P, et al. F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(17):14493-14501. doi:10.1609/aaai.v40i17.38466
BibTeX
@article{Bi2026,
author = {Bi, Hanbo and Yuan, Zhiqiang and Jia, Zexi and Zhang, Jiapei and Li, Chongyang and Luo, Peixiang and Deng, Ying and Duan, Xiaoyue and Zhang, Jinchao},
title = {F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {17},
pages = {14493--14501},
doi = {10.1609/aaai.v40i17.38466}
}
EUVIMED