EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG

Weitang Zhang, Lang Liu

Computers · 2026-07-20 · 15 (7) · 459

DOI: 10.3390/computers15070459 ↗

At a glance

Journal article in Computers, published online on 2026-07-20. Written by Weitang Zhang and 1 co-authors. 18 references. The full text is freely available (open access, CC BY 4.0).

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Addressing the resource constraints of a single NVIDIA RTX 5000 (16 GB) GPU, this applied study takes DeepSeek-LLM-7B-Base as the base model and systematically compares four parameter-efficient fine-tuning methods: LoRA, QLoRA, DoRA, and LoRA+. It also validates a Retrieval-Augmented Generation (RAG) architecture tailored for zero-tolerance engineering specifications. Experiments are conducted on a private construction process dataset. Theoretical analysis shows that the low-rank assumption of LoRA originates from the intrinsic dimensionality property of pre-trained models; LoRA+ adopts an asymmetric learning rate strategy (with the optimal ratio ηAηB = 0.05 determined via grid search), effectively solving the suboptimal training dynamics problem of standard LoRA caused by a uniform learning rate; DoRA decomposes weight updates into magnitude and direction components on a spherical manifold and a positive real manifold; RAG guarantees hallucination suppression through the conditional entropy inequality H(Y|Q,D,θ) ≤ H(Y|Q,θ). Experimental results demonstrate that LoRA+ outperforms other baseline methods in BLEU-4 (0.5609), ROUGE-L (0.5387), and PPL (2.1433), with a training time of 1.8 h and memory usage of 13.1 GB. After introducing RAG on top of LoRA+, BLEU-4 further improves to 0.5814, ROUGE-L to 0.5557, and the hallucination rate(HR) drops from 1.71% to 0.08%, achieving an Exact Match (EM) score of 0.2778 and high traceability (Recall@3 = 0.9961). This study provides a technical pathway and empirical evidence for deploying large models in the construction domain under resource-constrained conditions through the synergy of fine-tuning and RAG.

Cite

APA

Zhang, W., & Liu, L. (2026). Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG. Computers, 15(7), 459. https://doi.org/10.3390/computers15070459

Vancouver

Zhang W, Liu L. Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG. Computers. 2026;15(7):459. doi:10.3390/computers15070459

BibTeX
@article{Zhang2026,
  author = {Zhang, Weitang and Liu, Lang},
  title = {Application of LoRA+ in Fine-Tuning Large Models for Construction Process and Its Synergy with RAG},
  journal = {Computers},
  year = {2026},
  volume = {15},
  number = {7},
  pages = {459},
  doi = {10.3390/computers15070459}
}
Start for free →