EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Multimodal fusion for fine-grained plant classification: A visual–textual framework for accurate identification of solidago species

Garik Kazanjian, Abigail Loyson, Naren Kosaraju, Ally Atkins, Jahid Choton, Ivan Grijalva, Brian J. Spiesman, William Hsu, Venkat Margapuri

Smart Agricultural Technology · 2026-08 · 14 · 102045

DOI: 10.1016/j.atech.2026.102045 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Smart Agricultural Technology, 2026-08. Verfasst von Garik Kazanjian und 8 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 83 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: U.S. Department of Agriculture, National Institute of Food and Agriculture.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Goldenrod (Solidago spp.) is a genus of ecologically critical North American wildflowers that supports a diversity of pollinators, including bees, wasps, and and migratory insects such as Monarch butterflies. Yet species-level identification remains challenging due to extensive morphological similarity and overlapping distributions. Reliable automated classification would benefit ecological monitoring and biodiversity assessment, but visual models alone often struggle with fine-grained distinctions in natural imagery. This study introduces a multimodal framework that integrates image-based deep learning with text-derived ecological context to improve classification of 18 Solidago species.Images from the Global Biodiversity Information Facility (GBIF) are preprocessed using SAM2 and Grounding DINO to extract botanically relevant regions of interest. Five visual backbones such as VGG-19, ResNet-50, InceptionV3, Vision Transformer (ViT), and ConvNeXt are fine-tuned for species identification. In parallel, GPT-4o is used to generate structured botanical descriptions from GBIF metadata, which are then used to fine-tune a BERT-based classifier for textual inference. A late-fusion module integrates predictions from the visual and textual branches, leveraging complementary morphological and ecological cues.Across all architectures, multimodal fusion yields substantial improvements over image-only baselines, with ConvNeXt and ResNet achieving the highest overall accuracies (0.864 and 0.874) and strongest F1 scores (0.919 and 0.861). Extensive data analysis confirms that multimodal fusion consistently enhances discriminative power for morphologically similar taxa. To operationalize these findings, we also developed a cross-platform mobile application, enabling field-ready inference and real-time species exploration. Together, this work demonstrates that combining visual and contextual information provides a scalable and accurate pathway for identifying closely related plant species in natural environments.

Förderung

  • U.S. Department of Agriculture
  • National Institute of Food and Agriculture

Zitieren

APA

Kazanjian, G., Loyson, A., Kosaraju, N., Atkins, A., Choton, J., Grijalva, I., Spiesman, B. J., Hsu, W., & Margapuri, V. (2026). Multimodal fusion for fine-grained plant classification: A visual–textual framework for accurate identification of solidago species. Smart Agricultural Technology, 14, 102045. https://doi.org/10.1016/j.atech.2026.102045

Vancouver

Kazanjian G, Loyson A, Kosaraju N, Atkins A, Choton J, Grijalva I, et al. Multimodal fusion for fine-grained plant classification: A visual–textual framework for accurate identification of solidago species. Smart Agricultural Technology. 2026;14:102045. doi:10.1016/j.atech.2026.102045

BibTeX
@article{Kazanjian2026,
  author = {Kazanjian, Garik and Loyson, Abigail and Kosaraju, Naren and Atkins, Ally and Choton, Jahid and Grijalva, Ivan and Spiesman, Brian J. and Hsu, William and Margapuri, Venkat},
  title = {Multimodal fusion for fine-grained plant classification: A visual–textual framework for accurate identification of solidago species},
  journal = {Smart Agricultural Technology},
  year = {2026},
  volume = {14},
  pages = {102045},
  doi = {10.1016/j.atech.2026.102045}
}
Kostenlos starten →