EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Information-Theoretic Framework for Retrieval-Augmented Generation Systems

Semih Yumuşak

Electronics · 22.07.2025 · 14 (15) · 2925

DOI: 10.3390/electronics14152925 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 22.07.2025. Verfasst von Semih Yumuşak. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon Europe projects RAISE, UPCAST.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Retrieval-Augmented Generation (RAG) systems have emerged as a critical approach for enhancing large language models with external knowledge, yet the field lacks systematic theoretical analysis for understanding their fundamental characteristics and optimization principles. A novel information-theoretic approach for analyzing and optimizing RAG systems is introduced in this paper by modeling them as cascading information channel systems where each component (query encoding, retrieval, context integration, and generation) functions as a distinct information-theoretic channel with measurable capacity. Following established practices in information theory research, theoretical insights are evaluated through systematic experimentation on controlled synthetic datasets that enable precise manipulation of schema entropy and isolation of information flow dynamics. Through this controlled experimental approach, the following key theoretical insights are supported: (1) RAG performance is bounded by the minimum capacity across constituent channels, (2) the retrieval channel represents the primary information bottleneck, (3) errors propagate through channel-dependent mechanisms with specific interaction patterns, and (4) retrieval capacity is fundamentally limited by the minimum of embedding dimension and schema entropy. Both quantitative metrics for evaluating RAG systems and practical design principles for optimization are provided by the proposed approach. Retrieval improvements yield 58–85% performance gains and generation improvements yield 58–110% gains, substantially higher than context integration improvements (∼9%) and query encoding modifications, as shown by experimental results on controlled synthetic environments, supporting the theoretical approach. A systematic theoretical analysis for understanding RAG system dynamics is provided by this work, with real-world validation and practical implementation refinements representing natural next phases for this research.

Förderung

  • Horizon Europe projects RAISE
  • UPCAST

Zitieren

APA

Yumuşak, S. (2025). An Information-Theoretic Framework for Retrieval-Augmented Generation Systems. Electronics, 14(15), 2925. https://doi.org/10.3390/electronics14152925

Vancouver

Yumuşak S. An Information-Theoretic Framework for Retrieval-Augmented Generation Systems. Electronics. 2025;14(15):2925. doi:10.3390/electronics14152925

BibTeX
@article{Yumuak2025,
  author = {Yumuşak, Semih},
  title = {An Information-Theoretic Framework for Retrieval-Augmented Generation Systems},
  journal = {Electronics},
  year = {2025},
  volume = {14},
  number = {15},
  pages = {2925},
  doi = {10.3390/electronics14152925}
}
Kostenlos starten →