MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection
IEEE Transactions on Circuits and Systems for Video Technology · 2025-05 · 35 (5) · 5006-5021
DOI: 10.1109/tcsvt.2024.3524645 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-05. Verfasst von Taeheon Kim und 5 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 97 Literaturangaben. Gefördert von: Center for Applied Research in Artificial Intelligence, Defense Acquisition Program Administration (DAPA) and Agency for Defense Development.
Abstract (Originalsprache)
Multispectral pedestrian detection is attractive for around-the-clock applications due to the complementary information between RGB and thermal modalities. However, current models often fail to detect pedestrians in certain cases (e.g., thermal-obscured pedestrians), particularly due to the modality bias learned from statistically biased datasets. In this paper, we investigate how to mitigate modality bias in multispectral pedestrian detection using a Large Language Model (LLM). Accordingly, we design a Multispectral Chain-of-Thought (MSCoT) prompting strategy, which prompts the LLM to perform multispectral pedestrian detection. Moreover, we propose a novel Multispectral Chain-of-Thought Detection (MSCoTDet) framework that integrates MSCoT prompting into multispectral pedestrian detection. To this end, we design a Language-driven Multi-modal Fusion (LMF) strategy that enables fusing the outputs of MSCoT prompting with the detection results of vision-based multispectral pedestrian detection models. Extensive experiments validate that MSCoTDet effectively mitigates modality biases and improves multispectral pedestrian detection.
Förderung
- Center for Applied Research in Artificial Intelligence
- Defense Acquisition Program Administration (DAPA) and Agency for Defense Development
Zitieren
APA
Kim, T., Chung, S., Yeom, D., Yu, Y., Kim, H. G., & Ro, Y. M. (2025). MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection. IEEE Transactions on Circuits and Systems for Video Technology, 35(5), 5006-5021. https://doi.org/10.1109/tcsvt.2024.3524645
Vancouver
Kim T, Chung S, Yeom D, Yu Y, Kim HG, Ro YM. MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(5):5006-5021. doi:10.1109/tcsvt.2024.3524645
BibTeX
@article{Kim2025,
author = {Kim, Taeheon and Chung, Sangyun and Yeom, Damin and Yu, Youngjoon and Kim, Hak Gu and Ro, Yong Man},
title = {MSCoTDet: Language-Driven Multi-Modal Fusion for Improved Multispectral Pedestrian Detection},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2025},
volume = {35},
number = {5},
pages = {5006--5021},
doi = {10.1109/tcsvt.2024.3524645}
}
EUVIMED