Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification
Sensors · 07.11.2026 · 26 (21) · 7047
DOI: 10.3390/s26217047 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 07.11.2026. Verfasst von Hasibullah Mohmand und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Information Systems Engineering, Kocaeli University, Kocaeli 41001, Turkey. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Kocaeli University Technology Park, Kocaeli University Technopark.
Abstract (Originalsprache)
Disaster-related social media posts combine images and text, but the benefits of large language model (LLM)-derived representations for crisis classification remain uncertain. We evaluate three pretrained vision–language pipelines—CLIP, LLM2CLIP, and LLaVA-enhanced CLIP—on CrisisMMD informativeness, humanitarian, and image-derived damage-severity classification. Development experiments examine fusion and class-imbalance strategies; frozen-feature probes assess representation pathways, and selected configurations undergo five-seed held-out testing. CLIP achieves test macro-F1 of 92.32±0.55% and 87.52±1.09% for informativeness and humanitarian classification, respectively; development-selected LLaVA-enhanced CLIP achieves 64.06±0.98% for damage severity. Exact seed-blocked tests show no significant differences among pipeline families after Holm correction across nine contrasts (minimum adjusted p=0.5625). Frozen-feature probes show no consistent predictive advantage for LLM2CLIP-derived text under matched visual-feature controls. Because the released damage-severity partitions share tweet identifiers across splits, we also conduct five-seed tweet-disjoint retraining, yielding macro-F1 between 62.46% and 63.80% across the pipelines. A matched frozen CLIP control shows negligible change from adding the original tweet to the image input. Image–text retrieval, exploratory cross-event evaluation, and computational profiling characterize task-dependent alignment, transfer, and resource trade-offs. The study provides a disaster-domain comparison of established pipelines without isolating the effects of LLM-derived representations or establishing universal model-family superiority.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Information Systems Engineering, Kocaeli University, Kocaeli 41001, Turkey.
- Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli 41001, Turkey.
- EVSTEK Information Technology Company, Kocaeli 41275, Turkey.
ORCID-Profile: Hasibullah Mohmand, Zeynep Hilal Kilimci, Ayhan Küçükmanisa
Förderung
- Kocaeli University Technology Park
- Kocaeli University Technopark
Zitieren
APA
Mohmand, H., Kilimci, Z. H., & Küçükmanisa, A. (2026). Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification. Sensors, 26(21), 7047. https://doi.org/10.3390/s26217047
Vancouver
Mohmand H, Kilimci ZH, Küçükmanisa A. Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification. Sensors. 2026;26(21):7047. doi:10.3390/s26217047
BibTeX
@article{Mohmand2026,
author = {Mohmand, Hasibullah and Kilimci, Zeynep Hilal and Küçükmanisa, Ayhan},
title = {Systematic Evaluation of CLIP, LLM2CLIP, and LLaVA-Enhanced CLIP for Multimodal Disaster Classification},
journal = {Sensors},
year = {2026},
volume = {26},
number = {21},
pages = {7047},
doi = {10.3390/s26217047}
}
EUVIMED