MoFL: fine-tuning LLMs with mixture of fused LoRA experts
Complex & Intelligent Systems · 2026-12 · 12 (12)
DOI: 10.1007/s40747-026-02454-6 ↗Kurz gefasst
Fachartikel in der Zeitschrift Complex & Intelligent Systems, online erschienen am 04.08.2026. Verfasst von Shengchuan Lin und 4 weiteren Autor:innen. Die Arbeit nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Key Technologies Research and Development Program, Natural Science Basic Research Program of Shaanxi Province.
Abstract (Originalsprache)
Parameter-Efficient Fine-Tuning (PEFT) methods have enabled practical adaptation of Large Language Models (LLMs), yet integrating Mixture-of-Experts (MoE) architectures for multi-task scenarios introduces two critical challenges. First, existing MoE-based PEFT approaches overlook the need to balance commonalities and distinctions among diverse tasks, thereby limiting cross-task transferability. Second, prevailing routing strategies rely on discrete expert selection and gradient estimation techniques, which intermittently undermine the ability to consistently identify optimal routing pathways. To address these issues, we propose MoFL (Mixture of Fused LoRA), a hierarchical MoE architecture that incorporates Convergent Experts and Divergent Experts with fused low-rank parameters to jointly capture shared and task-specific knowledge. Complementing this design, a fully differentiable ensemble routing strategy is introduced, eliminating reliance on gradient estimation while enabling collaborative contributions from all experts during generation. Extensive experiments on the HuggingFace Open LLM benchmark demonstrate that MoFL achieves an average score of 52.45 while updating less than 2% of model parameters and consistently outperforms existing LoRA-based PEFT methods across diverse backbone LLMs. Notably, on the GSM8K mathematical reasoning benchmark, the mean score rises from 33.53 to 38.68, the most substantial improvement among all evaluated tasks.
Förderung
- National Natural Science Foundation of China
- Key Technologies Research and Development Program
- Natural Science Basic Research Program of Shaanxi Province
- Key Research and Development Program of Zhejiang Province
Zitieren
APA
Lin, S., Gao, D., Ma, Y., Hu, S., & Mu, D. (2026). MoFL: fine-tuning LLMs with mixture of fused LoRA experts. Complex & Intelligent Systems, 12(12). https://doi.org/10.1007/s40747-026-02454-6
Vancouver
Lin S, Gao D, Ma Y, Hu S, Mu D. MoFL: fine-tuning LLMs with mixture of fused LoRA experts. Complex & Intelligent Systems. 2026;12(12). doi:10.1007/s40747-026-02454-6
BibTeX
@article{Lin2026,
author = {Lin, Shengchuan and Gao, Dehong and Ma, Yufei and Hu, Shuangwei and Mu, Dejun},
title = {MoFL: fine-tuning LLMs with mixture of fused LoRA experts},
journal = {Complex & Intelligent Systems},
year = {2026},
volume = {12},
number = {12},
doi = {10.1007/s40747-026-02454-6}
}
EUVIMED