Context-Aware Retrieval Augmented Generation Using Similarity Validation to Handle Context Inconsistencies in Large Language Models
IEEE Access · 2025 · 13 · 170065-170080
DOI: 10.1109/access.2025.3614553 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Access, 2025. Verfasst von Enrico Collini und 3 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Ministry University of Research (MUR), project CAI4DSA of FAIR PE1.
Abstract (Originalsprache)
Large Language Models (LLMs) have transformed natural language processing by offering human-like responses. However, issues such as incorrect information (hallucinations) and errors in specific subject areas remain, especially in Retrieval Augmented Generation (RAG) systems. This study introduces a Context-Aware Retrieval Augmented Generation (CA-RAG), which simplifies the process by removing the need to separately find relevant chunks of a document. Instead, after dividing the document into chunks, both the question and chunks are directly given to the LLMs to produce answers. The method then focuses on improving the answers through additional post-processing, aiming to reduce errors and make the answers more relevant to the question. To evaluate the effectiveness of CA-RAG, two scenarios have been designed. Scenario 1 involved experiments using widely adopted and recognized benchmark datasets, such as TriviaQA, Natural Questions, AmbigQA and Stanford Question Answering Dataset (SQuAD). In this context, the proposed CA-RAG method, combined with similarity measure (either cosine similarity or dot product) between generated answers and chunks, achieved the highest F1-score in TriviaQA and AmbigQA. Scenario 2 tested CA-RAG robustness using a custom dataset comprising domain-specific and unstructured documents. Results from automated and manual evaluations revealed that CA-RAG with post-processing consistently outperformed traditional RAG. These findings highlight the critical role of post-processing techniques and similarity measures in improving the accuracy and relevance of generated answers. CA-RAG demonstrates strong potential as a reliable and versatile solution for Retrieval Augmented Generation tasks across diverse datasets and domains.
Förderung
- Ministry University of Research (MUR), project CAI4DSA of FAIR PE1
Zitieren
APA
Collini, E., Kurniadi, F. I., Nesi, P., & Pantaleo, G. (2025). Context-Aware Retrieval Augmented Generation Using Similarity Validation to Handle Context Inconsistencies in Large Language Models. IEEE Access, 13, 170065-170080. https://doi.org/10.1109/access.2025.3614553
Vancouver
Collini E, Kurniadi FI, Nesi P, Pantaleo G. Context-Aware Retrieval Augmented Generation Using Similarity Validation to Handle Context Inconsistencies in Large Language Models. IEEE Access. 2025;13:170065-170080. doi:10.1109/access.2025.3614553
BibTeX
@article{Collini2025,
author = {Collini, Enrico and Kurniadi, Felix Indra and Nesi, Paolo and Pantaleo, Gianni},
title = {Context-Aware Retrieval Augmented Generation Using Similarity Validation to Handle Context Inconsistencies in Large Language Models},
journal = {IEEE Access},
year = {2025},
volume = {13},
pages = {170065--170080},
doi = {10.1109/access.2025.3614553}
}
EUVIMED