EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Regulating Modality Utilization within Multimodal Fusion Networks

Saurav Singh iD, Eli Saber, Panos P Markopoulos iD, Jamison Heard iD

Sensors · 19.09.2024 · 24 (18) · 6054

DOI: 10.3390/s24186054 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 19.09.2024. Verfasst von Saurav Singh und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical & Microelectronic Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: United States Department of Defense, United States Air Force Office of Scientific Research, National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal fusion networks play a pivotal role in leveraging diverse sources of information for enhanced machine learning applications in aerial imagery. However, current approaches often suffer from a bias towards certain modalities, diminishing the potential benefits of multimodal data. This paper addresses this issue by proposing a novel modality utilization-based training method for multimodal fusion networks. The method aims to guide the network’s utilization on its input modalities, ensuring a balanced integration of complementary information streams, effectively mitigating the overutilization of dominant modalities. The method is validated on multimodal aerial imagery classification and image segmentation tasks, effectively maintaining modality utilization within ±10% of the user-defined target utilization and demonstrating the versatility and efficacy of the proposed method across various applications. Furthermore, the study explores the robustness of the fusion networks against noise in input modalities, a crucial aspect in real-world scenarios. The method showcases better noise robustness by maintaining performance amidst environmental changes affecting different aerial imagery sensing modalities. The network trained with 75.0% EO utilization achieves significantly better accuracy (81.4%) in noisy conditions (noise variance = 0.12) compared to traditional training methods with 99.59% EO utilization (73.7%). Additionally, it maintains an average accuracy of 85.0% across different noise levels, outperforming the traditional method’s average accuracy of 81.9%. Overall, the proposed approach presents a significant step towards harnessing the full potential of multimodal data fusion in diverse machine learning applications such as robotics, healthcare, satellite imagery, and defense applications.

Schlagwörter

Stichwörter der Autor:innen

aerial imagerydata fusionmodality utilizationmultimodalpermutation feature importance

Autor:innen und Institutionen

  • Department of Electrical & Microelectronic Engineering, Rochester Institute of Technology, Rochester, NY 14623, USA.
  • Department of Electrical & Computer Engineering and Department of Computer Science, The University of Texas at San Antonio, San Antonio, TX 78249, USA.

ORCID-Profile: Saurav Singh, Panos P Markopoulos, Jamison Heard

Förderung

  • United States Department of Defense
  • United States Air Force Office of Scientific Research
  • National Science Foundation
  • National Geospatial-Intelligence Agency
  • Air Force Office of Scientific Research (AFOSR)

Zitieren

APA

Singh, S., Saber, E., Markopoulos, P. P., & Heard, J. (2024). Regulating Modality Utilization within Multimodal Fusion Networks. Sensors, 24(18), 6054. https://doi.org/10.3390/s24186054

Vancouver

Singh S, Saber E, Markopoulos PP, Heard J. Regulating Modality Utilization within Multimodal Fusion Networks. Sensors. 2024;24(18):6054. doi:10.3390/s24186054

BibTeX
@article{Singh2024,
  author = {Singh, Saurav and Saber, Eli and Markopoulos, Panos P and Heard, Jamison},
  title = {Regulating Modality Utilization within Multimodal Fusion Networks},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {18},
  pages = {6054},
  doi = {10.3390/s24186054}
}
Kostenlos starten →