EUVIMEDEUVIMED entdecken
Crossref

Linguistic Hallucination for Text-Based Video Retrieval

Sheng Fang, Tiantian Dang, Shuhui Wang, Qingming Huang

IEEE Transactions on Circuits and Systems for Video Technology · 2024-10 · 34 (10) · 9692-9705

DOI: 10.1109/tcsvt.2024.3393843 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-10. Verfasst von Sheng Fang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 82 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Fundamental Research Funds for Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text-based video retrieval is a crucial technology for video and multimodal applications. Although in traditional Text-Video Retrieval caption-video pairs are supposed to be entirely relevant, there is still information missing in text when compared to the video content. In a specific application scenario of Text-Video Retrieval, where the given caption corresponds to only a segment of the target video, the challenge of aligning two modalities becomes particularly difficult. To address this issue, we introduce context information as an auxiliary to enrich text representation and enhance alignment. In this work, we propose an effective Linguistic Hallucination framework, which incorporates context captions during training and replaces them with hallucinated textual representations predicted from the source sentence at inference. Specific hallucination loss and consistency loss are designed to supervise the learning process. Besides, Curriculum Learning is introduced at both data-level and model-level, which makes the training procedure more stable and improves the retrieval performance simultaneously. Extensive comparison experiments and ablation studies on benchmark datasets demonstrate the effectiveness of our framework. Moreover, we also apply our proposed method to other cross-modal tasks and the promising experimental results prove its generalization ability. Our codes and datasets are available in https://github.com/silenceFS/Linguistic-Hallucination.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Fundamental Research Funds for Central Universities

Zitieren

APA

Fang, S., Dang, T., Wang, S., & Huang, Q. (2024). Linguistic Hallucination for Text-Based Video Retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 34(10), 9692-9705. https://doi.org/10.1109/tcsvt.2024.3393843

Vancouver

Fang S, Dang T, Wang S, Huang Q. Linguistic Hallucination for Text-Based Video Retrieval. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(10):9692-9705. doi:10.1109/tcsvt.2024.3393843

BibTeX
@article{Fang2024,
  author = {Fang, Sheng and Dang, Tiantian and Wang, Shuhui and Huang, Qingming},
  title = {Linguistic Hallucination for Text-Based Video Retrieval},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2024},
  volume = {34},
  number = {10},
  pages = {9692--9705},
  doi = {10.1109/tcsvt.2024.3393843}
}
Kostenlos starten →