Real-Time Recommendation Engines: A Stream-Native Five-Layer Architecture Integrating Kafka, Flink, Vector Retrieval, and LLM Augmentation
International Journal of Computer Sciences and Engineering · 30.06.2026 · 14 (6) · 15-25
DOI: 10.26438/ijcse.v14i6.7457 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Sciences and Engineering, online erschienen am 30.06.2026. Verfasst von Shubham Dandotiya und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Batch-trained recommendation models are refreshed on hourly or daily schedules, yet user interests shift on timescales of minutes. This mismatch between feature staleness and the pace of real-world behaviour leads to measurable quality degradation that algorithmic improvements alone cannot resolve. We have created a system that helps us make good recommendations to people. This system uses a few tools like Apache Kafka and Apache Flink. It also uses a store to keep track of information about people and things they like. We tested this system with a lot of information from a shopping website. The website has 100 million users. They do 2.5 billion things on the site. We found out that if we can show people things they might like really fast like in under five seconds, they are more likely to like our suggestions. This is an improvement from what we had before which took six hours to show new things. Our system is also really good at learning what people like and do not like. It can keep up with changes in what people like over time. We tried a different way to make our system even better and we found one that works really well. It helps us make better suggestions to people and it does not slow down our system too much. Some big companies, like Credit Karma, Netflix and TikTok have used our system. It has worked well for them. They have a lot of users. They need to be able to show them things they like really fast. Our system has helped them do that. We argue that federated streaming recommendation represents one of the most important open research problems in this area and characterize three unsolved challenges: communication efficiency, statistical heterogeneity, and temporal alignment.
Zitieren
APA
Dandotiya, S., & Sharma, P. K. (2026). Real-Time Recommendation Engines: A Stream-Native Five-Layer Architecture Integrating Kafka, Flink, Vector Retrieval, and LLM Augmentation. International Journal of Computer Sciences and Engineering, 14(6), 15-25. https://doi.org/10.26438/ijcse.v14i6.7457
Vancouver
Dandotiya S, Sharma PK. Real-Time Recommendation Engines: A Stream-Native Five-Layer Architecture Integrating Kafka, Flink, Vector Retrieval, and LLM Augmentation. International Journal of Computer Sciences and Engineering. 2026;14(6):15-25. doi:10.26438/ijcse.v14i6.7457
BibTeX
@article{Dandotiya2026,
author = {Dandotiya, Shubham and Sharma, Pharindra Kumar},
title = {Real-Time Recommendation Engines: A Stream-Native Five-Layer Architecture Integrating Kafka, Flink, Vector Retrieval, and LLM Augmentation},
journal = {International Journal of Computer Sciences and Engineering},
year = {2026},
volume = {14},
number = {6},
pages = {15--25},
doi = {10.26438/ijcse.v14i6.7457}
}
EUVIMED