ROBUSTNESS-AWARE MODEL COMPRESSION FOR MULTIMODAL PERCEPTION UNDER CHALLENGING CONDITIONS
AI and Data Science Journal · 31.12.2024 · 1 (1) · 103-111
DOI: 10.61784/adsj4042 ↗Kurz gefasst
Fachartikel in der Zeitschrift AI and Data Science Journal, online erschienen am 31.12.2024. Verfasst von FengYuan Zhang und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Compressed multimodal perception models are usually validated on clean benchmarks, yet they are deployed where visibility is low, sensors are occluded and measurements are noisy. This paper studies RGB-thermal semantic segmentation on the MSRS road-scene benchmark and asks three questions: which predictions break after compression under challenging conditions, how compression, calibration and multimodal reliability cues can repair them, and how accuracy, hard-scenario performance and real latency trade off. We first show that 4-bit quantization and structured pruning remove 2.8-4.9 mIoU points on clean data but up to 7.4 points under unseen corruptions, and that the resulting harmful prediction flips concentrate on low-margin pixels, object boundaries and rare classes. We then propose RAMC, a robustness-aware multimodal compression framework that combines condition-aware Hessian sensitivity and minimax integer programming for mixed-precision post-training quantization with condition-aware Taylor pruning, paired clean-to-degraded distillation and fusion-gate consistency, and we analyze post-compression confidence calibration. At a tight budget of 24 BitOPs per MAC, RAMC improves night mIoU by 1.2 points and mean corrupted mIoU by 0.4 points over HAWQ while reducing harmful flips by 17%. With 75% of the hidden channels removed, it improves mean corrupted mIoU by 1.2 points over distillation; at 50% pruning it exceeds the uncompressed model in clean, night and mean corrupted mIoU while running 1.24-1.47 times faster in ONNX Runtime. Distilled students become underconfident, and a single temperature fitted on held-out degraded data nearly halves their calibration error on unseen corruptions.
Zitieren
APA
Zhang, F., & Yang, J. (2024). ROBUSTNESS-AWARE MODEL COMPRESSION FOR MULTIMODAL PERCEPTION UNDER CHALLENGING CONDITIONS. AI and Data Science Journal, 1(1), 103-111. https://doi.org/10.61784/adsj4042
Vancouver
Zhang F, Yang J. ROBUSTNESS-AWARE MODEL COMPRESSION FOR MULTIMODAL PERCEPTION UNDER CHALLENGING CONDITIONS. AI and Data Science Journal. 2024;1(1):103-111. doi:10.61784/adsj4042
BibTeX
@article{Zhang2024,
author = {Zhang, FengYuan and Yang, JieSi},
title = {ROBUSTNESS-AWARE MODEL COMPRESSION FOR MULTIMODAL PERCEPTION UNDER CHALLENGING CONDITIONS},
journal = {AI and Data Science Journal},
year = {2024},
volume = {1},
number = {1},
pages = {103--111},
doi = {10.61784/adsj4042}
}
EUVIMED