EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections

Thomas Smits, Melvin Wevers

Digital Scholarship in the Humanities · 31.08.2023 · 38 (3) · 1267-1280

DOI: 10.1093/llc/fqad008 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Digital Scholarship in the Humanities, online erschienen am 15.03.2023. Verfasst von Thomas Smits und 1 weiteren Autor:innen. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 50 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Until recently, most research in the Digital Humanities (DH) was monomodal, meaning that the object of analysis was either textual or visual. Seeking to integrate multimodality theory into the DH, this article demonstrates that recently developed multimodal deep learning models, such as Contrastive Language Image Pre-training (CLIP), offer new possibilities to explore and analyze image–text combinations at scale. These models, which are trained on image and text pairs, can be applied to a wide range of text-to-image, image-to-image, and image-to-text prediction tasks. Moreover, multimodal models show high accuracy in zero-shot classification, i.e. predicting unseen categories across heterogeneous datasets. Based on three exploratory case studies, we argue that this zero-shot capability opens up the way for a multimodal turn in DH research. Moreover, multimodal models allow scholars to move past the artificial separation of text and images that was dominant in the field and analyze multimodal meaning at scale. However, we also need to be aware of the specific (historical) bias of multimodal deep learning that stems from biases in the training data used to train these models.

Zitieren

APA

Smits, T., & Wevers, M. (2023). A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections. Digital Scholarship in the Humanities, 38(3), 1267-1280. https://doi.org/10.1093/llc/fqad008

Vancouver

Smits T, Wevers M. A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections. Digital Scholarship in the Humanities. 2023;38(3):1267-1280. doi:10.1093/llc/fqad008

BibTeX
@article{Smits2023,
  author = {Smits, Thomas and Wevers, Melvin},
  title = {A multimodal turn in Digital Humanities. Using contrastive machine learning models to explore, enrich, and analyze digital visual historical collections},
  journal = {Digital Scholarship in the Humanities},
  year = {2023},
  volume = {38},
  number = {3},
  pages = {1267--1280},
  doi = {10.1093/llc/fqad008}
}
Kostenlos starten →