EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

SIM‐Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification

Youcef Sklab, Hanane Ariouat, Eric Chenin, Edi Prifti, Jean‐Daniel Zucker

IET Computer Vision · 2025-01 · 19 (1)

DOI: 10.1049/cvi2.70036 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IET Computer Vision, online erschienen am 09.07.2025. Verfasst von Youcef Sklab und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Agence Nationale de la Recherche.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We introduce the shape‐image multimodal network (SIM‐Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel‐to‐point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture‐based and geometric features for enhanced classification performance. SIM‐Net is particularly well‐suited for the classification of digitised herbarium specimens—a task made challenging by heterogeneous backgrounds, nonplant elements, and occlusions that compromise conventional image‐based models. To address these issues, SIM‐Net employs a segmentation‐based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet‐based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM‐Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F‐score. It also surpasses several transformer‐based state‐of‐the‐art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.

Förderung

  • Agence Nationale de la Recherche

Zitieren

APA

Sklab, Y., Ariouat, H., Chenin, E., Prifti, E., & Zucker, J. (2025). SIM‐Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification. IET Computer Vision, 19(1). https://doi.org/10.1049/cvi2.70036

Vancouver

Sklab Y, Ariouat H, Chenin E, Prifti E, Zucker J. SIM‐Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification. IET Computer Vision. 2025;19(1). doi:10.1049/cvi2.70036

BibTeX
@article{Sklab2025,
  author = {Sklab, Youcef and Ariouat, Hanane and Chenin, Eric and Prifti, Edi and Zucker, Jean‐Daniel},
  title = {SIM‐Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification},
  journal = {IET Computer Vision},
  year = {2025},
  volume = {19},
  number = {1},
  doi = {10.1049/cvi2.70036}
}
Kostenlos starten →