EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition

Minwei Chen, Yajun Liu, Zenghui Zhang iD, Weiwei Guo iD

Sensors · 24.07.2024 · 24 (15) · 4803

DOI: 10.3390/s24154803 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 24.07.2024. Verfasst von Minwei Chen und 3 weiteren Autor:innen. Beteiligte Einrichtung: Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, Shanghai 200240, China. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: the National Natural Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Robust object detection in complex environments, poor visual conditions, and open scenarios presents significant technical challenges in autonomous driving. These challenges necessitate the development of advanced fusion methods for millimeter-wave (mmWave) radar point cloud data and visual images. To address these issues, this paper proposes a radar–camera robust fusion network (RCRFNet), which leverages self-supervised learning and open-set recognition to effectively utilise the complementary information from both sensors. Specifically, the network uses matched radar–camera data through a frustum association approach to generate self-supervised signals, enhancing network training. The integration of global and local depth consistencies between radar point clouds and visual images, along with image features, helps construct object class confidence levels for detecting unknown targets. Additionally, these techniques are combined with a multi-layer feature extraction backbone and a multimodal feature detection head to achieve robust object detection. Experiments on the nuScenes public dataset demonstrate that RCRFNet outperforms state-of-the-art (SOTA) methods, particularly in conditions of low visual visibility and when detecting unknown class objects.

Schlagwörter

Stichwörter der Autor:innen

autonomous drivingopen-set recognitionradar–camera fusionself-supervised learningtarget detection

Autor:innen und Institutionen

  • Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, Shanghai 200240, China.
  • Center of Digital Innovation, Tongji University, Shanghai 200092, China.

ORCID-Profile: Zenghui Zhang, Weiwei Guo

Förderung

  • the National Natural Science Foundation of China

Zitieren

APA

Chen, M., Liu, Y., Zhang, Z., & Guo, W. (2024). RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition. Sensors, 24(15), 4803. https://doi.org/10.3390/s24154803

Vancouver

Chen M, Liu Y, Zhang Z, Guo W. RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition. Sensors. 2024;24(15):4803. doi:10.3390/s24154803

BibTeX
@article{Chen2024,
  author = {Chen, Minwei and Liu, Yajun and Zhang, Zenghui and Guo, Weiwei},
  title = {RCRFNet: Enhancing Object Detection with Self-Supervised Radar–Camera Fusion and Open-Set Recognition},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {15},
  pages = {4803},
  doi = {10.3390/s24154803}
}
Kostenlos starten →