Lightweight Large Language Model Distillation for Technical Question Answering with Verified Citations
American Journal of Machine Learning · 2025
DOI: 10.71465/ajml3459 ↗Kurz gefasst
Fachartikel in der Zeitschrift American Journal of Machine Learning, 2025. Verfasst von Betty Lopez und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
The proliferation of Large Language Models (LLMs) has revolutionized the field of natural language processing, particularly in the domain of question answering (QA). However, the deployment of these massive models in resource-constrained environments remains a significant challenge due to their high computational memory and latency requirements. Furthermore, in technical domains such as software engineering, medical diagnosis, and legal analysis, the hallucination of non-existent facts or references poses a critical reliability risk. This paper introduces a novel framework for distilling the capabilities of massive LLMs (Teacher models) into lightweight architectures (Student models) specifically optimized for Technical Question Answering with Verified Citations (TQA-VC). We propose a dual-objective loss function that balances semantic fidelity with citation accuracy, ensuring that the Student model not only generates coherent technical responses but also correctly attributes information to provided source documents. Our approach integrates a citation verification module during the distillation process, penalizing the Student for unsupported claims. Experimental results on proprietary and public technical datasets demonstrate that our 7-billion parameter Student model achieves 94% of the performance of a 70-billion parameter Teacher model while reducing inference latency by a factor of four and significantly improving citation precision compared to standard distillation techniques.
Zitieren
APA
Lopez, B., & Gonzalez, W. (2025). Lightweight Large Language Model Distillation for Technical Question Answering with Verified Citations. American Journal of Machine Learning. https://doi.org/10.71465/ajml3459
Vancouver
Lopez B, Gonzalez W. Lightweight Large Language Model Distillation for Technical Question Answering with Verified Citations. American Journal of Machine Learning. 2025. doi:10.71465/ajml3459
BibTeX
@article{Lopez2025,
author = {Lopez, Betty and Gonzalez, William},
title = {Lightweight Large Language Model Distillation for Technical Question Answering with Verified Citations},
journal = {American Journal of Machine Learning},
year = {2025},
doi = {10.71465/ajml3459}
}
EUVIMED