EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification

Hasibullah Mohmand iD, Zeynep Hilal Kilimci iD, Ayhan Küçükmanisa iD

Sensors · 07.11.2026 · 26 (21) · 7047

DOI: 10.3390/s26217047 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 07.11.2026. Verfasst von Hasibullah Mohmand und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Information Systems Engineering, Kocaeli University, Kocaeli 41001, Turkey. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Kocaeli University Technology Park, Kocaeli University Technopark.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Disaster-related social media posts combine images and text, but the benefits of large language model (LLM)-derived representations for crisis classification remain uncertain. We evaluate three pretrained vision–language pipelines—CLIP, LLM2CLIP, and LLaVA-enhanced CLIP—on CrisisMMD informativeness, humanitarian, and image-derived damage-severity classification. Development experiments examine fusion and class-imbalance strategies; frozen-feature probes assess representation pathways, and selected configurations undergo five-seed held-out testing. CLIP achieves test macro-F1 of 92.32±0.55% and 87.52±1.09% for informativeness and humanitarian classification, respectively; development-selected LLaVA-enhanced CLIP achieves 64.06±0.98% for damage severity. Exact seed-blocked tests show no significant differences among pipeline families after Holm correction across nine contrasts (minimum adjusted p=0.5625). Frozen-feature probes show no consistent predictive advantage for LLM2CLIP-derived text under matched visual-feature controls. Because the released damage-severity partitions share tweet identifiers across splits, we also conduct five-seed tweet-disjoint retraining, yielding macro-F1 between 62.46% and 63.80% across the pipelines. A matched frozen CLIP control shows negligible change from adding the original tweet to the image input. Image–text retrieval, exploratory cross-event evaluation, and computational profiling characterize task-dependent alignment, transfer, and resource trade-offs. The study provides a disaster-domain comparison of established pipelines without isolating the effects of LLM-derived representations or establishing universal model-family superiority.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsHumansAlgorithmsClassification Algorithms

Stichwörter der Autor:innen

CLIPCrisisMMDLLM2CLIPcross-modal fusionmultimodal disaster classificationvision–language models

Autor:innen und Institutionen

  • Department of Information Systems Engineering, Kocaeli University, Kocaeli 41001, Turkey.
  • Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli 41001, Turkey.
  • EVSTEK Information Technology Company, Kocaeli 41275, Turkey.

ORCID-Profile: Hasibullah Mohmand, Zeynep Hilal Kilimci, Ayhan Küçükmanisa

Förderung

  • Kocaeli University Technology Park
  • Kocaeli University Technopark

Zitieren

APA

Mohmand, H., Kilimci, Z. H., & Küçükmanisa, A. (2026). Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification. Sensors, 26(21), 7047. https://doi.org/10.3390/s26217047

Vancouver

Mohmand H, Kilimci ZH, Küçükmanisa A. Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification. Sensors. 2026;26(21):7047. doi:10.3390/s26217047

BibTeX
@article{Mohmand2026,
  author = {Mohmand, Hasibullah and Kilimci, Zeynep Hilal and Küçükmanisa, Ayhan},
  title = {Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification},
  journal = {Sensors},
  year = {2026},
  volume = {26},
  number = {21},
  pages = {7047},
  doi = {10.3390/s26217047}
}
Kostenlos starten →