MMIFEvol: Towards Evolutionary Multimodal Instruction Following
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (31) · 26206-26214
DOI: 10.1609/aaai.v40i31.39824 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Haoyu Wang und 7 weiteren Autor:innen.
Abstract (Originalsprache)
Multimodal Instruction Following serves as a fundamental capability of multimodal language models, involving accurate comprehension and execution of user-provided instructions. However, existing multimodal instruction-following datasets and benchmarks face the shortcomings outlined below: (a) Lack of Difficulty Stratification, they collect diverse instruction categories but neglect the stratification of difficulty levels across these categories, which leads to overlap, bias, and low interpretability. (b) Lack of Fine-Grained Metrics, they conflate the model's ability to ``solve tasks" and ``follow constraints" into a single metric, which fails to accurately reflect its instruction-following capability. (c) Lack of Multi-Task Instructions, they overlook the fact that real-world user instructions often consist of multiple combined tasks. This paper proposes MMIFEvol, a framework for multimodal instruction evolving and benchmarking. First, we define the essential components of a carefully curated multimodal instruction set and establish corresponding difficulty levels, based on which we synthesize diverse instruction data. Next, we decouple the evaluation criteria for the instruction following into three different metrics to construct a high-quality benchmark and assess existing models. Experimental results demonstrate that current models still struggle with following complex instructions, while fine-tuning using MMIFEvol data effectively improves models' responsiveness to multimodal instructions.
Zitieren
APA
Wang, H., Jiang, S., Zhu, X., Chen, Y., Meng, X., Wei, J., Wang, Y., & Xiao, Y. (2026). MMIFEvol: Towards Evolutionary Multimodal Instruction Following. Proceedings of the AAAI Conference on Artificial Intelligence, 40(31), 26206-26214. https://doi.org/10.1609/aaai.v40i31.39824
Vancouver
Wang H, Jiang S, Zhu X, Chen Y, Meng X, Wei J, et al. MMIFEvol: Towards Evolutionary Multimodal Instruction Following. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(31):26206-26214. doi:10.1609/aaai.v40i31.39824
BibTeX
@article{Wang2026,
author = {Wang, Haoyu and Jiang, Sihang and Zhu, Xiangru and Chen, Yuyan and Meng, Xiaojun and Wei, Jiansheng and Wang, Yitong and Xiao, Yanghua},
title = {MMIFEvol: Towards Evolutionary Multimodal Instruction Following},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {31},
pages = {26206--26214},
doi = {10.1609/aaai.v40i31.39824}
}
EUVIMED