Multimodal fine-grained grocery product recognition using image and OCR text
Machine Vision and Applications · 2024-07 · 35 (4)
DOI: 10.1007/s00138-024-01549-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Machine Vision and Applications, online erschienen am 07.06.2024. Verfasst von Tobias Pettersson und 2 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 98 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Smart Industry Sweden, Swedish National Infrastructure for Computing.
Abstract (Originalsprache)
Automatic recognition of grocery products can be used to improve customer flow at checkouts and reduce labor costs and store losses. Product recognition is, however, a challenging task for machine learning-based solutions due to the large number of products and their variations in appearance. In this work, we tackle the challenge of fine-grained product recognition by first extracting a large dataset from a grocery store containing products that are only differentiable by subtle details. Then, we propose a multimodal product recognition approach that uses product images with extracted OCR text from packages to improve fine-grained recognition of grocery products. We evaluate several image and text models separately and then combine them using different multimodal models of varying complexities. The results show that image and textual information complement each other in multimodal models and enable a classifier with greater recognition performance than unimodal models, especially when the number of training samples is limited. Therefore, this approach is suitable for many different scenarios in which product recognition is used to further improve recognition performance. The dataset can be found at https://github.com/Tubbias/finegrainocr .
Förderung
- Smart Industry Sweden
- Swedish National Infrastructure for Computing
Zitieren
APA
Pettersson, T., Riveiro, M., & Löfström, T. (2024). Multimodal fine-grained grocery product recognition using image and OCR text. Machine Vision and Applications, 35(4). https://doi.org/10.1007/s00138-024-01549-9
Vancouver
Pettersson T, Riveiro M, Löfström T. Multimodal fine-grained grocery product recognition using image and OCR text. Machine Vision and Applications. 2024;35(4). doi:10.1007/s00138-024-01549-9
BibTeX
@article{Pettersson2024,
author = {Pettersson, Tobias and Riveiro, Maria and Löfström, Tuwe},
title = {Multimodal fine-grained grocery product recognition using image and OCR text},
journal = {Machine Vision and Applications},
year = {2024},
volume = {35},
number = {4},
doi = {10.1007/s00138-024-01549-9}
}
EUVIMED