EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Autonomous AI Agents for Apache Flink Pipeline Management on Kubernetes

Jyothish Sreedharan

International Journal of Computational and Experimental Science and Engineering · 05.03.2026 · 12 (1)

DOI: 10.22399/ijcesen.4998 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computational and Experimental Science and Engineering, online erschienen am 05.03.2026. Verfasst von Jyothish Sreedharan. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 10 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Modern companies struggle to maintain data pipelines that process millions of events per second while meeting strict performance requirements. Traditional methods use fixed configurations and manual fixes, which fail when workloads change unexpectedly. This paper presents an AI-powered system that automatically manages Apache Flink pipelines on Kubernetes. The system uses machine learning to predict problems before they occur, recover from failures automatically, and optimize resource usage continuously. The system was evaluated using two publicly available benchmark datasets: the NYC Taxi Trip Record dataset adapted for streaming scenarios and the Yahoo Cloud Serving Benchmark dataset. Tests show the AI-driven approach significantly reduces service violations, substantially cuts recovery time, and lowers infrastructure costs compared to manual management while maintaining better performance. The system uses three AI agents working together where the prediction agent forecasts problems ahead of time with high accuracy using a neural network that processes multiple metrics continuously, the recovery agent detects failures rapidly using isolation forests, autoencoders, and long short-term memory networks, and the optimization agent adjusts resources dynamically based on workload patterns using reinforcement learning. Together, these agents enable the system to operate autonomously, dramatically reducing manual interventions and operational overhead.

Zitieren

APA

Sreedharan, J. (2026). Autonomous AI Agents for Apache Flink Pipeline Management on Kubernetes. International Journal of Computational and Experimental Science and Engineering, 12(1). https://doi.org/10.22399/ijcesen.4998

Vancouver

Sreedharan J. Autonomous AI Agents for Apache Flink Pipeline Management on Kubernetes. International Journal of Computational and Experimental Science and Engineering. 2026;12(1). doi:10.22399/ijcesen.4998

BibTeX
@article{Sreedharan2026,
  author = {Sreedharan, Jyothish},
  title = {Autonomous AI Agents for Apache Flink Pipeline Management on Kubernetes},
  journal = {International Journal of Computational and Experimental Science and Engineering},
  year = {2026},
  volume = {12},
  number = {1},
  doi = {10.22399/ijcesen.4998}
}
Kostenlos starten →