EUVIMEDEUVIMED entdecken
Crossref

Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation

Rongyu Zhang, Yulin Luo, Jiaming Liu, Huanrui Yang, Zhen Dong, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Yuan Du, Shanghang Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (15) · 16812-16820

DOI: 10.1609/aaai.v38i15.29622 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Rongyu Zhang und 10 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Mixture-of-Experts (MoE) approach has demonstrated outstanding scalability in multi-task learning including low-level upstream tasks such as concurrent removal of multiple adverse weather effects. However, the conventional MoE architecture with parallel Feed Forward Network (FFN) experts leads to significant parameter and computational overheads that hinder its efficient deployment. In addition, the naive MoE linear router is suboptimal in assigning task-specific features to multiple experts which limits its further scalability. In this work, we propose an efficient MoE architecture with weight sharing across the experts. Inspired by the idea of linear feature modulation (FM), our architecture implicitly instantiates multiple experts via learnable activation modulations on a single shared expert block. The proposed Feature Modulated Expert (FME) serves as a building block for the novel Mixture-of-Feature-Modulation-Experts (MoFME) architecture, which can scale up the number of experts with low overhead. We further propose an Uncertainty-aware Router (UaR) to assign task-specific features to different FM modules with well-calibrated weights. This enables MoFME to effectively learn diverse expert functions for multiple tasks. The conducted experiments on the multi-deweather task show that our MoFME outperforms the state-of-the-art in the image restoration quality by 0.1-0.2 dB while saving more than 74% of parameters and 20% inference time over the conventional MoE counterpart. Experiments on the downstream segmentation and classification tasks further demonstrate the generalizability of MoFME to real open-world applications.

Zitieren

APA

Zhang, R., Luo, Y., Liu, J., Yang, H., Dong, Z., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., Du, Y., & Zhang, S. (2024). Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15), 16812-16820. https://doi.org/10.1609/aaai.v38i15.29622

Vancouver

Zhang R, Luo Y, Liu J, Yang H, Dong Z, Gudovskiy D, et al. Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(15):16812-16820. doi:10.1609/aaai.v38i15.29622

BibTeX
@article{Zhang2024,
  author = {Zhang, Rongyu and Luo, Yulin and Liu, Jiaming and Yang, Huanrui and Dong, Zhen and Gudovskiy, Denis and Okuno, Tomoyuki and Nakata, Yohei and Keutzer, Kurt and Du, Yuan and Zhang, Shanghang},
  title = {Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {15},
  pages = {16812--16820},
  doi = {10.1609/aaai.v38i15.29622}
}
Kostenlos starten →