EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type

Romy Müller, Marcel Dürschmidt, Julian Ullrich, Carsten Knoll, Sascha Weber, Steffen Seitz

Applied Sciences · 21.03.2024 · 14 (6) · 2648

DOI: 10.3390/app14062648 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 21.03.2024. Verfasst von Romy Müller und 5 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Research Foundation (DFG).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deep neural networks are powerful image classifiers but do they attend to similar image areas as humans? While previous studies have investigated how this similarity is shaped by technological factors, little is known about the role of factors that affect human attention. Therefore, we investigated the interactive effects of task and image characteristics. We varied the intentionality of the tasks used to elicit human attention maps (i.e., spontaneous gaze, gaze-pointing, manual area selection). Moreover, we varied the type of image to be categorized (i.e., singular objects, indoor scenes consisting of object arrangements, landscapes without distinct objects). The human attention maps generated in this way were compared to the attention maps of a convolutional neural network (CNN) as revealed by a method of explainable artificial intelligence (Grad-CAM). The influence of human tasks strongly depended on image type: for objects, human manual selection produced attention maps that were most similar to CNN, while the specific eye movement task had little impact. For indoor scenes, spontaneous gaze produced the least similarity, while for landscapes, similarity was equally low across all human tasks. Our results highlight the importance of taking human factors into account when comparing the attention of humans and CNN.

Förderung

  • German Research Foundation (DFG)

Zitieren

APA

Müller, R., Dürschmidt, M., Ullrich, J., Knoll, C., Weber, S., & Seitz, S. (2024). Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type. Applied Sciences, 14(6), 2648. https://doi.org/10.3390/app14062648

Vancouver

Müller R, Dürschmidt M, Ullrich J, Knoll C, Weber S, Seitz S. Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type. Applied Sciences. 2024;14(6):2648. doi:10.3390/app14062648

BibTeX
@article{Mller2024,
  author = {Müller, Romy and Dürschmidt, Marcel and Ullrich, Julian and Knoll, Carsten and Weber, Sascha and Seitz, Steffen},
  title = {Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type},
  journal = {Applied Sciences},
  year = {2024},
  volume = {14},
  number = {6},
  pages = {2648},
  doi = {10.3390/app14062648}
}
Kostenlos starten →