EUVIMEDEUVIMED entdecken
Crossref

When Social Media Images Need Words: Measuring Context Gap and Fusion Tax in Crisis Image Captioning

Amanda Hughes, Yuhao Bao

Proceedings of the International ISCRAM Conference · 22.05.2026 · 23

DOI: 10.59297/4bar0c32 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the International ISCRAM Conference, online erschienen am 22.05.2026. Verfasst von Amanda Hughes und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Crisis images on social media can be difficult to interpret at scale, and many carry meaning embedded in text or symbols (e.g., radar screenshots, evacuation notices). This limits vision-only captioning for situational awareness. We quantify a central trade-off in multimodal captioning: adding post text can reduce omission-driven ambiguity (the Context Gap), but it can also introduce text-driven errors (the Fusion Tax). Using 204 high-priority image–post pairs from CrisisFACTS, we compare Vision-only and Vision + Text captioning across Gemini 2.0 Flash, Qwen2.5-VL, and BLIP. We find that post text improves accuracy for Gemini and Qwen largely by reducing misidentification and scene-type errors, while sometimes amplifying hallucinated (unsupported) details. BLIP, however, does not reliably fuse modalities in our setup. When post text is provided, it often collapses into simple text echoing rather than producing image-grounded captions. We discuss implications for multimodal fusion in crisis informatics and outline next steps for image-type evaluation and routing.

Zitieren

APA

Hughes, A., & Bao, Y. (2026). When Social Media Images Need Words: Measuring Context Gap and Fusion Tax in Crisis Image Captioning. Proceedings of the International ISCRAM Conference, 23. https://doi.org/10.59297/4bar0c32

Vancouver

Hughes A, Bao Y. When Social Media Images Need Words: Measuring Context Gap and Fusion Tax in Crisis Image Captioning. Proceedings of the International ISCRAM Conference. 2026;23. doi:10.59297/4bar0c32

BibTeX
@article{Hughes2026,
  author = {Hughes, Amanda and Bao, Yuhao},
  title = {When Social Media Images Need Words: Measuring Context Gap and Fusion Tax in Crisis Image Captioning},
  journal = {Proceedings of the International ISCRAM Conference},
  year = {2026},
  volume = {23},
  doi = {10.59297/4bar0c32}
}
Kostenlos starten →