EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains

Alexander Sergeev, Valeriya Buzmakova, Evgeny Kotelnikov

Journal on Interactive Systems · 09.07.2026 · 17 (1) · 619-638

DOI: 10.5753/jis.2026.7338 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal on Interactive Systems, online erschienen am 09.07.2026. Verfasst von Alexander Sergeev und 2 weiteren Autor:innen. Die Arbeit nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The study addresses the problem of underutilization of the capabilities of modern Multimodal Large Language Models in the Digital Humanities. We propose an architecture for a scalable and adaptive multimodal virtual intelligent assistant based on the Retrieval-Augmented Generation approach. The architecture was tested on heterogeneous data in Russian from three subject domains: history, literature, and botany. Three modality processing strategies were used: text-only, images-only, and combined (text and images). In the retrieval experiments, full-text search, semantic text search, image search, and hybrid variants were compared. The most effective combination proved to be full-text search, semantic text search using the KaLM-Embedding model, and image search using the MetaCLIP 2 model. In the generation experiments, six open-weight Multimodal LLMs were compared using the LLM-as-Judge approach. The most performative model for the combined modality was Gemma 3, while for the text-only modality it was Qwen-3-VL Instruct. When working with images all models demonstrated relatively low performance. The study confirms the practical applicability of the proposed architecture and identifies candidate models for developing virtual intelligent assistants capable of working with complex multimodal databases in digital humanities projects.

Zitieren

APA

Sergeev, A., Buzmakova, V., & Kotelnikov, E. (2026). Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains. Journal on Interactive Systems, 17(1), 619-638. https://doi.org/10.5753/jis.2026.7338

Vancouver

Sergeev A, Buzmakova V, Kotelnikov E. Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains. Journal on Interactive Systems. 2026;17(1):619-638. doi:10.5753/jis.2026.7338

BibTeX
@article{Sergeev2026,
  author = {Sergeev, Alexander and Buzmakova, Valeriya and Kotelnikov, Evgeny},
  title = {Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains},
  journal = {Journal on Interactive Systems},
  year = {2026},
  volume = {17},
  number = {1},
  pages = {619--638},
  doi = {10.5753/jis.2026.7338}
}
Kostenlos starten →