EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Adaptive Audiovisual Fusion Method Based on Prediction Confidence for Fine Granularity Bird Species Recognition

Xinliang Xu, Qiming Liu, Xin Wen, Heng Zhao, Zhenhao Wang, Chong Wang

Applied Sciences · 20.05.2026 · 16 (10) · 5113

DOI: 10.3390/app16105113 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 20.05.2026. Verfasst von Xinliang Xu und 5 weiteren Autor:innen. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Harbin Power Supply Company, State Grid Heilongjiang Electric Power Co., Ltd..

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To address the inherent limitations of single-modality approaches in fine-grained bird species recognition, this paper proposes an adaptive audiovisual fusion method based on prediction confidence. The proposed framework comprises three core components: an image classification branch, an audio classification branch, and a confidence–adaptive fusion module. The image branch employs EfficientNet-B3 to extract fine-grained visual features through compound scaling and squeeze-and-excitation (SE) attention. The audio branch utilizes ResNet-50 to classify Mel spectrograms converted from bird vocalizations, incorporating a dense sampling inference strategy to fully exploit complete audio information. For multimodal integration, a confidence–adaptive fusion strategy is introduced that jointly considers information entropy and probability gap to dynamically assess the reliability of each modality’s prediction, thereby assigning fusion weights at the sample level without any additional trainable parameters. Experiments on the SSW60 multimodal bird recognition dataset show that the image branch achieves a Top-1 accuracy of 91.55%, outperforming ResNet-50 (89.75%) and VGG-16 (83.81%); the audio branch reaches 68.20%, surpassing AST (63.29%) and VGG-16 (53.48%); and the fused model attains 95.30% Top-1 accuracy, a 3.75 percentage-point improvement over the image-only baseline and a 0.21 percentage-point gain over the learning-based TMC fusion baseline without introducing any trainable parameters, confirming the effectiveness of the proposed method.

Förderung

  • Harbin Power Supply Company, State Grid Heilongjiang Electric Power Co., Ltd.

Zitieren

APA

Xu, X., Liu, Q., Wen, X., Zhao, H., Wang, Z., & Wang, C. (2026). An Adaptive Audiovisual Fusion Method Based on Prediction Confidence for Fine Granularity Bird Species Recognition. Applied Sciences, 16(10), 5113. https://doi.org/10.3390/app16105113

Vancouver

Xu X, Liu Q, Wen X, Zhao H, Wang Z, Wang C. An Adaptive Audiovisual Fusion Method Based on Prediction Confidence for Fine Granularity Bird Species Recognition. Applied Sciences. 2026;16(10):5113. doi:10.3390/app16105113

BibTeX
@article{Xu2026,
  author = {Xu, Xinliang and Liu, Qiming and Wen, Xin and Zhao, Heng and Wang, Zhenhao and Wang, Chong},
  title = {An Adaptive Audiovisual Fusion Method Based on Prediction Confidence for Fine Granularity Bird Species Recognition},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {10},
  pages = {5113},
  doi = {10.3390/app16105113}
}
Kostenlos starten →