EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Analysis of Threshold Similarity Effect on Dense Retrieval in Retrieval-Augmented Generation Systems Using the SQuAD Dataset

Yehezkiel Fienathan Bolo Dadi, Cendra Putra

Journal of Artificial Intelligence and Data Science · 31.07.2026 · 2 (2) · 105-115

DOI: 10.67561/jaids.v2i2.5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Artificial Intelligence and Data Science, online erschienen am 31.07.2026. Verfasst von Yehezkiel Fienathan Bolo Dadi und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-Augmented Generation (RAG), which combines text generation and document retrieval, is one of the more useful methods for maintaining big language model outputs based on actual, verifiable sources rather than the model's own stored knowledge. However, what is initially extracted from the retrieval stage has a significant impact on how well a RAG system responds to a query. Using the Stanford Question Answering Dataset (SQuAD) as the testbed, this study especially examines how the similarity threshold employed during dense retrieval affects both retrieval quality and, later on in the pipeline, the quality of the final answer. The multi-qa-mpnet-base-dot-v1 embedding model was used to encapsulate context passages and questions, which were then compared using cosine similarity. The answer generator was LLaMA 3.1 8B, which was accessed using the Groq API. Fifty questions selected from the SQuAD validation set were used to test eleven threshold values, ranging from 0.0 to 1.0 in steps of 0.1. The quality of retrieval was evaluated by the measures Recall, Precision, and Mean Reciprocal Rank (MRR). The generated answers were assessed by means of Exact Match (EM) and F1 Measure. The baseline performance (at 0.0 threshold) had F1 at 0.827 and EM at 0.72, while for the 0.3 threshold it became F1 at 0.848 and EM at 0.78. Once past the 0.5 threshold, all metrics showed a sharp drop. Overall, the conclusions that can be drawn from this experiment are that threshold-based filtering of context significantly improves the performance of a RAG model.

Zitieren

APA

Dadi, Y. F. B., & Putra, C. (2026). Analysis of Threshold Similarity Effect on Dense Retrieval in Retrieval-Augmented Generation Systems Using the SQuAD Dataset. Journal of Artificial Intelligence and Data Science, 2(2), 105-115. https://doi.org/10.67561/jaids.v2i2.5

Vancouver

Dadi YFB, Putra C. Analysis of Threshold Similarity Effect on Dense Retrieval in Retrieval-Augmented Generation Systems Using the SQuAD Dataset. Journal of Artificial Intelligence and Data Science. 2026;2(2):105-115. doi:10.67561/jaids.v2i2.5

BibTeX
@article{Dadi2026,
  author = {Dadi, Yehezkiel Fienathan Bolo and Putra, Cendra},
  title = {Analysis of Threshold Similarity Effect on Dense Retrieval in Retrieval-Augmented Generation Systems Using the SQuAD Dataset},
  journal = {Journal of Artificial Intelligence and Data Science},
  year = {2026},
  volume = {2},
  number = {2},
  pages = {105--115},
  doi = {10.67561/jaids.v2i2.5}
}
Kostenlos starten →