EUVIMEDEUVIMED entdecken
Open Access · CC BY-ND 4.0

Early Fusion of Visual and Ingredient Representations for Multimodal Food Classification

Navira Rahma Salsabila, Adela Regita Azzahra, Fitri Utaminingrum, Barlian Henryranu Prasetio

Jurnal Online Informatika · 24.04.2026 · 11 (1) · 60-69

DOI: 10.15575/join.v11i1.1725 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Jurnal Online Informatika, online erschienen am 24.04.2026. Verfasst von Navira Rahma Salsabila und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Identifying the most appropriate food dish based on available kitchen ingredients remains a practical yet challenging task in everyday life. To address this, this study specifically aims to develop an intelligent food classification system using a multimodal approach. We propose a multimodal food classification method that performs early fusion by combining visual and textual features extracted using the Contrastive Language–Image Pretraining (CLIP) model. Features from food images and ingredient lists are fused and classified through a two-layer multilayer perceptron. The model is evaluated on the Recipes5k dataset with 4,826 samples across 101 food categories. Results show that the proposed multimodal model achieves 91.32% accuracy, outperforming text-only (85.65%) and image-only (57.26%) baselines. The main contribution of this work lies in demonstrating the effectiveness of early fusion for combining cross-modal representations in food classification. Unlike prior methods, our model supports flexible inference with either text or image input, enabling practical real-world applications. These findings highlight the potential of multimodal learning for food recommendation systems, offering both accuracy and contextual relevance beyond unimodal approaches.

Zitieren

APA

Salsabila, N. R., Azzahra, A. R., Utaminingrum, F., & Prasetio, B. H. (2026). Early Fusion of Visual and Ingredient Representations for Multimodal Food Classification. Jurnal Online Informatika, 11(1), 60-69. https://doi.org/10.15575/join.v11i1.1725

Vancouver

Salsabila NR, Azzahra AR, Utaminingrum F, Prasetio BH. Early Fusion of Visual and Ingredient Representations for Multimodal Food Classification. Jurnal Online Informatika. 2026;11(1):60-69. doi:10.15575/join.v11i1.1725

BibTeX
@article{Salsabila2026,
  author = {Salsabila, Navira Rahma and Azzahra, Adela Regita and Utaminingrum, Fitri and Prasetio, Barlian Henryranu},
  title = {Early Fusion of Visual and Ingredient Representations for Multimodal Food Classification},
  journal = {Jurnal Online Informatika},
  year = {2026},
  volume = {11},
  number = {1},
  pages = {60--69},
  doi = {10.15575/join.v11i1.1725}
}
Kostenlos starten →