EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

QuantFT-VL: Harmonizing Quantization and LoRA for Efficient Mobile Vision–Language Model Fine-Tuning

Fangyuan Jin, Hui Lin, Lu Zhang, Yiwei Chen

Algorithms · 04.05.2026 · 19 (5) · 364

DOI: 10.3390/a19050364 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Algorithms, online erschienen am 04.05.2026. Verfasst von Fangyuan Jin und 3 weiteren Autor:innen. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Funding project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vision–language models (VLMs) are increasingly deployed in resource-constrained environments, yet efficient fine-tuning remains challenging because post-training quantization often degrades the effectiveness of low-rank adaptation. This paper revisits that mismatch in the context of MobileVLM1.7B and presents QuantFT-VL, a novel initialization strategy following the quantization phase to seamlessly align with the LoRA technique. The key idea is to initialize LoRA using a low-rank approximation of the quantization residual instead of the default zero-initialization used in QLoRA-style pipelines. After quantizing a pretrained weight matrix W into Q, we compute the residual W − Q and use truncated singular value decomposition to initialize the LoRA factors (A and B) so that the starting adapted weight Q + ABT better matches the full-precision model. This residual-aware initialization reduces the discrepancy introduced by quantization and leads to faster and more stable optimization. Experiments on six standard VLM benchmarks show that QuantFT-VL consistently improves over QLoRA and recovers performance close to or better than full-precision LoRA in the best setting. On two RTX 3090 GPUs, QuantFT-VL improves the average benchmark score by 3.27 percentage points over QLoRA while preserving the memory and speed advantages of quantized fine-tuning.

Förderung

  • Funding project

Zitieren

APA

Jin, F., Lin, H., Zhang, L., & Chen, Y. (2026). QuantFT-VL: Harmonizing Quantization and LoRA for Efficient Mobile Vision–Language Model Fine-Tuning. Algorithms, 19(5), 364. https://doi.org/10.3390/a19050364

Vancouver

Jin F, Lin H, Zhang L, Chen Y. QuantFT-VL: Harmonizing Quantization and LoRA for Efficient Mobile Vision–Language Model Fine-Tuning. Algorithms. 2026;19(5):364. doi:10.3390/a19050364

BibTeX
@article{Jin2026,
  author = {Jin, Fangyuan and Lin, Hui and Zhang, Lu and Chen, Yiwei},
  title = {QuantFT-VL: Harmonizing Quantization and LoRA for Efficient Mobile Vision–Language Model Fine-Tuning},
  journal = {Algorithms},
  year = {2026},
  volume = {19},
  number = {5},
  pages = {364},
  doi = {10.3390/a19050364}
}
Kostenlos starten →