EUVIMEDObjevte EUVIMED
Open access · CC BY 4.0

Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications

Arunabh Bora, Heriberto Cuayáhuitl

Machine Learning and Knowledge Extraction · 2024-10-18 · 6 (4) · 2355-2374

DOI: 10.3390/make6040116 ↗

At a glance

Journal article in Machine Learning and Knowledge Extraction, published online on 2024-10-18. Written by Arunabh Bora and 1 co-authors. Cited 67 times so far (Crossref); 47 references. The full text is freely available (open access, CC BY 4.0).

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Artificial Intelligence (AI) has the potential to revolutionise the medical and healthcare sectors. AI and related technologies could significantly address some supply-and-demand challenges in the healthcare system, such as medical AI assistants, chatbots and robots. This paper focuses on tailoring LLMs to medical data utilising a Retrieval-Augmented Generation (RAG) database to evaluate their performance in a computationally resource-constrained environment. Existing studies primarily focus on fine-tuning LLMs on medical data, but this paper combines RAG and fine-tuned models and compares them against base models using RAG or only fine-tuning. Open-source LLMs (Flan-T5-Large, LLaMA-2-7B, and Mistral-7B) are fine-tuned using the medical datasets Meadow-MedQA and MedMCQA. Experiments are reported for response generation and multiple-choice question answering. The latter uses two distinct methodologies: Type A, as standard question answering via direct choice selection; and Type B, as language generation and probability confidence score generation of choices available. Results in the medical domain revealed that Fine-tuning and RAG are crucial for improved performance, and that methodology Type A outperforms Type B.

Cite

APA

Bora, A., & Cuayáhuitl, H. (2024). Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications. Machine Learning and Knowledge Extraction, 6(4), 2355-2374. https://doi.org/10.3390/make6040116

Vancouver

Bora A, Cuayáhuitl H. Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications. Machine Learning and Knowledge Extraction. 2024;6(4):2355-2374. doi:10.3390/make6040116

BibTeX
@article{Bora2024,
  author = {Bora, Arunabh and Cuayáhuitl, Heriberto},
  title = {Systematic Analysis of Retrieval-Augmented Generation-Based LLMs for Medical Chatbot Applications},
  journal = {Machine Learning and Knowledge Extraction},
  year = {2024},
  volume = {6},
  number = {4},
  pages = {2355--2374},
  doi = {10.3390/make6040116}
}
Start for free →