EUVIMEDEUVIMED entdecken
Crossref

Speculative Retrieval-Augmented Generation for Cost-Efficient Large Language Model Inference

Dr. Sumalatha P, Manoj Kumar

International Scientific Journal of Engineering and Management · 06.05.2026 · 05 (05) · 1-2

DOI: 10.55041/isjem07213 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Scientific Journal of Engineering and Management, online erschienen am 06.05.2026. Verfasst von Dr. Sumalatha P und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

- This work focuses on the two crucial bottlenecks in Retrieval-Augmented Generation (RAG): high inference latency and expensive computation cost. RAG enhances the factual correctness of Large Language Models (LLMs) by utilizing external knowledge sources; However, the auto-To achieve a good recall, the system applies a two-way retrieval mechanism: Dense retrieval: This module uses BAAI/bge-small-en-v1.5 to create a 384-dimensional embedding space, where queries and documents are mapped. The embeddings are stored in a FAISS IndexFlatL2. Sparse retrieval: This module uses TF-IDF vectorization to capture an exact keyword match. Reranking: A simple heuristic mechanism of matching substrings of the query terms in the documents selected to perform the final step of ranking them for prompt introduce a novel speculative RAG framework that combines hybrid retrieval with speculative decoding. Our framework employs hybrid dense vector search (BGE-small) and sparse keyword search (TF-IDF) for enhanced recall followed by a lightweight re-ranker model. A compact draft model (TinyLlama-1.1B) makes tentative future token generation, then a larger verifier model (Mistral-7B) checks the correctness of the candidate token. Empirical experiments demonstrate 33% and 29% improvement in inference latency and token reduction, respectively, compared to the baseline of a typical 7B model, while retaining 94% accuracy on factoid tasks. Key Words: Retrieval-Augmented Generation, Speculative Decoding, Large Language Models, Inference Optimization, Hybrid Retrieval

Zitieren

APA

P, D. S., & Kumar, M. (2026). Speculative Retrieval-Augmented Generation for Cost-Efficient Large Language Model Inference. International Scientific Journal of Engineering and Management, 05(05), 1-2. https://doi.org/10.55041/isjem07213

Vancouver

P DS, Kumar M. Speculative Retrieval-Augmented Generation for Cost-Efficient Large Language Model Inference. International Scientific Journal of Engineering and Management. 2026;05(05):1-2. doi:10.55041/isjem07213

BibTeX
@article{P2026,
  author = {P, Dr. Sumalatha and Kumar, Manoj},
  title = {Speculative Retrieval-Augmented Generation for Cost-Efficient Large Language Model Inference},
  journal = {International Scientific Journal of Engineering and Management},
  year = {2026},
  volume = {05},
  number = {05},
  pages = {1--2},
  doi = {10.55041/isjem07213}
}
Kostenlos starten →