EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems

Róbert Lakatos, Péter Pollner, András Hajdu, Tamás Joó

Machine Learning and Knowledge Extraction · 2025-02-10 · 7 (1) · 15

DOI: 10.3390/make7010015 ↗

At a glance

Journal article in Machine Learning and Knowledge Extraction, published online on 2025-02-10. Written by Róbert Lakatos and 3 co-authors. Cited 46 times so far (Crossref); 34 references. The full text is freely available (open access, CC BY 4.0). Funded by: Ministry of Culture and Innovation of Hungary from the National Research, Development, and Innovation Fund, National Research, Development and Innovation Office of Hungary.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Generative large language models (LLMs) have revolutionized the development of knowledge-based systems, enabling new possibilities in applications like ChatGPT, Bing, and Gemini. Two key strategies for domain adaptation in these systems are Domain-Specific Fine-Tuning (DFT) and Retrieval-Augmented Generation (RAG). In this study, we evaluate the performance of RAG and DFT on several LLM architectures, including GPT-J-6B, OPT-6.7B, LLaMA, and LLaMA-2. We use the ROUGE, BLEU, and METEOR scores to evaluate the performance of the models. We also measure the performance of the models with our own designed cosine similarity-based Coverage Score (CS). Our results, based on experiments across multiple datasets, show that RAG-based systems consistently outperform those fine-tuned with DFT. Specifically, RAG models outperform DFT by an average of 17% in ROUGE, 13% in BLEU, and 36% in CS. At the same time, DFT achieves only a modest advantage in METEOR, suggesting slightly better creative capabilities. We also highlight the challenges of integrating RAG with DFT, as such integration can lead to performance degradation. Furthermore, we propose a simplified RAG-based architecture that maximizes efficiency and reduces hallucination, underscoring the advantages of RAG in building reliable, domain-adapted knowledge systems.

Funding

  • Ministry of Culture and Innovation of Hungary from the National Research, Development, and Innovation Fund
  • National Research, Development and Innovation Office of Hungary

Cite

APA

Lakatos, R., Pollner, P., Hajdu, A., & Joó, T. (2025). Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems. Machine Learning and Knowledge Extraction, 7(1), 15. https://doi.org/10.3390/make7010015

Vancouver

Lakatos R, Pollner P, Hajdu A, Joó T. Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems. Machine Learning and Knowledge Extraction. 2025;7(1):15. doi:10.3390/make7010015

BibTeX
@article{Lakatos2025,
  author = {Lakatos, Róbert and Pollner, Péter and Hajdu, András and Joó, Tamás},
  title = {Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2025},
  volume = {7},
  number = {1},
  pages = {15},
  doi = {10.3390/make7010015}
}
Start for free →