EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

MoFL: fine-tuning LLMs with mixture of fused LoRA experts

Shengchuan Lin, Dehong Gao, Yufei Ma, Shuangwei Hu, Dejun Mu

Complex & Intelligent Systems · 2026-12 · 12 (12)

DOI: 10.1007/s40747-026-02454-6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Complex & Intelligent Systems, online erschienen am 04.08.2026. Verfasst von Shengchuan Lin und 4 weiteren Autor:innen. Die Arbeit nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Key Technologies Research and Development Program, Natural Science Basic Research Program of Shaanxi Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Parameter-Efficient Fine-Tuning (PEFT) methods have enabled practical adaptation of Large Language Models (LLMs), yet integrating Mixture-of-Experts (MoE) architectures for multi-task scenarios introduces two critical challenges. First, existing MoE-based PEFT approaches overlook the need to balance commonalities and distinctions among diverse tasks, thereby limiting cross-task transferability. Second, prevailing routing strategies rely on discrete expert selection and gradient estimation techniques, which intermittently undermine the ability to consistently identify optimal routing pathways. To address these issues, we propose MoFL (Mixture of Fused LoRA), a hierarchical MoE architecture that incorporates Convergent Experts and Divergent Experts with fused low-rank parameters to jointly capture shared and task-specific knowledge. Complementing this design, a fully differentiable ensemble routing strategy is introduced, eliminating reliance on gradient estimation while enabling collaborative contributions from all experts during generation. Extensive experiments on the HuggingFace Open LLM benchmark demonstrate that MoFL achieves an average score of 52.45 while updating less than 2% of model parameters and consistently outperforms existing LoRA-based PEFT methods across diverse backbone LLMs. Notably, on the GSM8K mathematical reasoning benchmark, the mean score rises from 33.53 to 38.68, the most substantial improvement among all evaluated tasks.

Förderung

  • National Natural Science Foundation of China
  • Key Technologies Research and Development Program
  • Natural Science Basic Research Program of Shaanxi Province
  • Key Research and Development Program of Zhejiang Province

Zitieren

APA

Lin, S., Gao, D., Ma, Y., Hu, S., & Mu, D. (2026). MoFL: fine-tuning LLMs with mixture of fused LoRA experts. Complex & Intelligent Systems, 12(12). https://doi.org/10.1007/s40747-026-02454-6

Vancouver

Lin S, Gao D, Ma Y, Hu S, Mu D. MoFL: fine-tuning LLMs with mixture of fused LoRA experts. Complex & Intelligent Systems. 2026;12(12). doi:10.1007/s40747-026-02454-6

BibTeX
@article{Lin2026,
  author = {Lin, Shengchuan and Gao, Dehong and Ma, Yufei and Hu, Shuangwei and Mu, Dejun},
  title = {MoFL: fine-tuning LLMs with mixture of fused LoRA experts},
  journal = {Complex & Intelligent Systems},
  year = {2026},
  volume = {12},
  number = {12},
  doi = {10.1007/s40747-026-02454-6}
}
Kostenlos starten →