Multi-modal Content-Based Recommendation Approach Using Vision-Language Models
JOIV : International Journal on Informatics Visualization · 31.07.2026 · 10 (4) · 1880
DOI: 10.62527/joiv.10.4.5961 ↗Kurz gefasst
Fachartikel in der Zeitschrift JOIV : International Journal on Informatics Visualization, online erschienen am 31.07.2026. Verfasst von Dongjoon Suh und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).
Abstract (Originalsprache)
Recommender systems play a pivotal role in helping users discover desired content amidst an overwhelmingly vast sea of multimedia information, mitigating the severe choice overload experienced on modern digital platforms. Traditional content-based filtering algorithms typically measure item similarity by directly comparing text-based attributes or relying on explicit keywords to generate recommendations. However, these conventional approaches face significant limitations in fully grasping the subtle contextual nuances, emotional undertones, and multi-sensory nature of modern digital content such as movies and television series. While recent advancements have successfully integrated Large Language Models (LLMs) to enrich the semantic representation of textual metadata, they inherently ignore the crucial visual elements—such as cinematography, color palettes, and lighting—that profoundly influence user preferences. This study proposes a novel Multi-modal Content-Based Filtering approach that leverages Vision-Language Models (VLMs) to simultaneously process and effectively fuse unstructured text data (plots and synopses) with high-dimensional visual data (video segments and sampled frames). By projecting both modalities into a shared latent semantic space and fusing their respective embeddings through a dedicated deep learning fusion module, the proposed system successfully captures both the deep narrative structure and the implicit visual atmosphere of the content. Experimental results conducted on a comprehensively merged dataset of MovieLens metadata and corresponding official movie trailers confirm that the proposed multi-modal architecture significantly outperforms traditional keyword-based methods and text-only LLM-based approaches across key evaluation metrics, demonstrating substantial improvements in overall recommendation accuracy, model robustness, and catalog diversity.
Zitieren
APA
Suh, D., Seo, T., & Chung, K. (2026). Multi-modal Content-Based Recommendation Approach Using Vision-Language Models. JOIV : International Journal on Informatics Visualization, 10(4), 1880. https://doi.org/10.62527/joiv.10.4.5961
Vancouver
Suh D, Seo T, Chung K. Multi-modal Content-Based Recommendation Approach Using Vision-Language Models. JOIV : International Journal on Informatics Visualization. 2026;10(4):1880. doi:10.62527/joiv.10.4.5961
BibTeX
@article{Suh2026,
author = {Suh, Dongjoon and Seo, Tae-Young and Chung, Kyungyong},
title = {Multi-modal Content-Based Recommendation Approach Using Vision-Language Models},
journal = {JOIV : International Journal on Informatics Visualization},
year = {2026},
volume = {10},
number = {4},
pages = {1880},
doi = {10.62527/joiv.10.4.5961}
}
EUVIMED