Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering
Journal of Imaging · 23.02.2024 · 10 (3) · 56
DOI: 10.3390/jimaging10030056 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 23.02.2024. Verfasst von Qiwen Lu und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer and Information Engineering, Henan University, Kaifeng 475001, China. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Key Scientific and Technological Project of Henan Province of China.
Abstract (Originalsprache)
Language bias stands as a noteworthy concern in visual question answering (VQA), wherein models tend to rely on spurious correlations between questions and answers for prediction. This prevents the models from effectively generalizing, leading to a decrease in performance. In order to address this bias, we propose a novel modality fusion collaborative de-biasing algorithm (CoD). In our approach, bias is considered as the model’s neglect of information from a particular modality during prediction. We employ a collaborative training approach to facilitate mutual modeling between different modalities, achieving efficient feature fusion and enabling the model to fully leverage multimodal knowledge for prediction. Our experiments on various datasets, including VQA-CP v2, VQA v2, and VQA-VS, using different validation strategies, demonstrate the effectiveness of our approach. Notably, employing a basic baseline model resulted in an accuracy of 60.14% on VQA-CP v2.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Computer and Information Engineering, Henan University, Kaifeng 475001, China.
Förderung
- Key Scientific and Technological Project of Henan Province of China
Zitieren
APA
Lu, Q., Chen, S., & Zhu, X. (2024). Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering. Journal of Imaging, 10(3), 56. https://doi.org/10.3390/jimaging10030056
Vancouver
Lu Q, Chen S, Zhu X. Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering. Journal of Imaging. 2024;10(3):56. doi:10.3390/jimaging10030056
BibTeX
@article{Lu2024,
author = {Lu, Qiwen and Chen, Shengbo and Zhu, Xiaoke},
title = {Collaborative Modality Fusion for Mitigating Language Bias in Visual Question Answering},
journal = {Journal of Imaging},
year = {2024},
volume = {10},
number = {3},
pages = {56},
doi = {10.3390/jimaging10030056}
}
EUVIMED