EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems

Róbert Lakatos, Péter Pollner, András Hajdu, Tamás Joó

Machine Learning and Knowledge Extraction · 10.02.2025 · 7 (1) · 15

DOI: 10.3390/make7010015 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Machine Learning and Knowledge Extraction, online erschienen am 10.02.2025. Verfasst von Róbert Lakatos und 3 weiteren Autor:innen. Die Arbeit wurde bisher 46-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Culture and Innovation of Hungary from the National Research, Development, and Innovation Fund, National Research, Development and Innovation Office of Hungary.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Generative large language models (LLMs) have revolutionized the development of knowledge-based systems, enabling new possibilities in applications like ChatGPT, Bing, and Gemini. Two key strategies for domain adaptation in these systems are Domain-Specific Fine-Tuning (DFT) and Retrieval-Augmented Generation (RAG). In this study, we evaluate the performance of RAG and DFT on several LLM architectures, including GPT-J-6B, OPT-6.7B, LLaMA, and LLaMA-2. We use the ROUGE, BLEU, and METEOR scores to evaluate the performance of the models. We also measure the performance of the models with our own designed cosine similarity-based Coverage Score (CS). Our results, based on experiments across multiple datasets, show that RAG-based systems consistently outperform those fine-tuned with DFT. Specifically, RAG models outperform DFT by an average of 17% in ROUGE, 13% in BLEU, and 36% in CS. At the same time, DFT achieves only a modest advantage in METEOR, suggesting slightly better creative capabilities. We also highlight the challenges of integrating RAG with DFT, as such integration can lead to performance degradation. Furthermore, we propose a simplified RAG-based architecture that maximizes efficiency and reduces hallucination, underscoring the advantages of RAG in building reliable, domain-adapted knowledge systems.

Förderung

  • Ministry of Culture and Innovation of Hungary from the National Research, Development, and Innovation Fund
  • National Research, Development and Innovation Office of Hungary

Zitieren

APA

Lakatos, R., Pollner, P., Hajdu, A., & Joó, T. (2025). Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems. Machine Learning and Knowledge Extraction, 7(1), 15. https://doi.org/10.3390/make7010015

Vancouver

Lakatos R, Pollner P, Hajdu A, Joó T. Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems. Machine Learning and Knowledge Extraction. 2025;7(1):15. doi:10.3390/make7010015

BibTeX
@article{Lakatos2025,
  author = {Lakatos, Róbert and Pollner, Péter and Hajdu, András and Joó, Tamás},
  title = {Investigating the Performance of Retrieval-Augmented Generation and Domain-Specific Fine-Tuning for the Development of AI-Driven Knowledge-Based Systems},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2025},
  volume = {7},
  number = {1},
  pages = {15},
  doi = {10.3390/make7010015}
}
Kostenlos starten →