EUVIMEDEUVIMED entdecken
Crossref

Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations

Ruoyu Chen, Jingzhi Li, Hua Zhang, Changchong Sheng, Li Liu, Xiaochun Cao

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.11.2023 · 19 (6) · 1-22

DOI: 10.1145/3563039 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 12.07.2023. Verfasst von Ruoyu Chen und 5 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 83 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Beijing Natural Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, we have witnessed substantial success using the deep neural network in many tasks. Although there still exist concerns about the explainability of decision making, it is beneficial for users to discern the defects in the deployed deep models. Existing explainable models either provide the image-level visualization of attention weights or generate textual descriptions as post hoc justifications. Different from existing models, in this article we propose a new interpretation method that explains the image similarity models by salience maps and attribute words. Our interpretation model contains visual salience maps generation and the counterfactual explanation generation. The former has two branches: global identity relevant region discovery and multi-attribute semantic region discovery. The first branch aims to capture the visual evidence supporting the similarity score, which is achieved by computing counterfactual feature maps. The second branch aims to discover semantic regions supporting different attributes, which helps to understand which attributes in an image might change the similarity score. Then, by fusing visual evidence from two branches, we can obtain the salience maps indicating important response evidence. The latter will generate the attribute words that best explain the similarity using the proposed erasing model. The effectiveness of our model is evaluated on the classical face verification task. Experiments conducted on two benchmarks—VGGFace2 and Celeb-A—demonstrate that our model can provide convincing interpretable explanations for the similarity. Moreover, our algorithm can be applied to evidential learning cases, such as finding the most characteristic attributes in a set of face images, and we verify its effectiveness on the VGGFace2 dataset.

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China
  • Beijing Natural Science Foundation

Zitieren

APA

Chen, R., Li, J., Zhang, H., Sheng, C., Liu, L., & Cao, X. (2023). Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations. ACM Transactions on Multimedia Computing, Communications, and Applications, 19(6), 1-22. https://doi.org/10.1145/3563039

Vancouver

Chen R, Li J, Zhang H, Sheng C, Liu L, Cao X. Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations. ACM Transactions on Multimedia Computing, Communications, and Applications. 2023;19(6):1-22. doi:10.1145/3563039

BibTeX
@article{Chen2023,
  author = {Chen, Ruoyu and Li, Jingzhi and Zhang, Hua and Sheng, Changchong and Liu, Li and Cao, Xiaochun},
  title = {Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2023},
  volume = {19},
  number = {6},
  pages = {1--22},
  doi = {10.1145/3563039}
}
Kostenlos starten →