Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type
Applied Sciences · 21.03.2024 · 14 (6) · 2648
DOI: 10.3390/app14062648 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 21.03.2024. Verfasst von Romy Müller und 5 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Research Foundation (DFG).
Abstract (Originalsprache)
Deep neural networks are powerful image classifiers but do they attend to similar image areas as humans? While previous studies have investigated how this similarity is shaped by technological factors, little is known about the role of factors that affect human attention. Therefore, we investigated the interactive effects of task and image characteristics. We varied the intentionality of the tasks used to elicit human attention maps (i.e., spontaneous gaze, gaze-pointing, manual area selection). Moreover, we varied the type of image to be categorized (i.e., singular objects, indoor scenes consisting of object arrangements, landscapes without distinct objects). The human attention maps generated in this way were compared to the attention maps of a convolutional neural network (CNN) as revealed by a method of explainable artificial intelligence (Grad-CAM). The influence of human tasks strongly depended on image type: for objects, human manual selection produced attention maps that were most similar to CNN, while the specific eye movement task had little impact. For indoor scenes, spontaneous gaze produced the least similarity, while for landscapes, similarity was equally low across all human tasks. Our results highlight the importance of taking human factors into account when comparing the attention of humans and CNN.
Förderung
- German Research Foundation (DFG)
Zitieren
APA
Müller, R., Dürschmidt, M., Ullrich, J., Knoll, C., Weber, S., & Seitz, S. (2024). Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type. Applied Sciences, 14(6), 2648. https://doi.org/10.3390/app14062648
Vancouver
Müller R, Dürschmidt M, Ullrich J, Knoll C, Weber S, Seitz S. Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type. Applied Sciences. 2024;14(6):2648. doi:10.3390/app14062648
BibTeX
@article{Mller2024,
author = {Müller, Romy and Dürschmidt, Marcel and Ullrich, Julian and Knoll, Carsten and Weber, Sascha and Seitz, Steffen},
title = {Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {6},
pages = {2648},
doi = {10.3390/app14062648}
}
EUVIMED