EUVIMEDEUVIMED entdecken
Crossref

Intelligent Orchestration : AI-Driven Optimization of Cloud Data Pipelines in Heterogeneous and Serverless Environments

Srinivasa Rao Seetala

International Journal of Scientific Research in Science Engineering and Technology · 01.05.2023 · 876

DOI: 10.32628/ijsrset23107111 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Scientific Research in Science Engineering and Technology, online erschienen am 22.05.2023. Verfasst von Srinivasa Rao Seetala. Die Arbeit nennt 15 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Modern cloud platforms host large-scale data pipelines that power analytics, machine learning, and real-time decision systems across industries ranging from healthcare to finance. As data volumes grow exponentially and workloads become increasingly heterogeneous spanning batch analytics, streaming ingestion, and GPU-accelerated ML tasks traditional rule-based scheduling and static resource provisioning mechanisms struggle to satisfy evolving performance, cost, and reliability constraints. These legacy approaches often rely on fixed heuristics, overprovisioned clusters, or reactive scaling, leading to suboptimal utilization, latency bottlenecks, and inflated operational expenses. This article explores AI-driven approaches to optimizing cloud data pipelines by embedding machine learning into workflow scheduling, predictive autoscaling, anomaly detection, and cross-layer resource orchestration. Grounded in foundational distributed systems research such as the MapReduce paradigm introduced by Jeffrey Dean and Sanjay Ghemawat, as well as the DAG-based dataflow model of Apache Spark, and extended by modern serverless execution engines and global optimization frameworks like AGORA, we propose an architectural blueprint that leverages telemetry-driven feedback loops, predictive modeling, and reinforcement learning to enable adaptive, cost-aware decision-making. By synthesizing key research developments from 2000 to 2022, this work outlines how intelligent scheduling policies, workload forecasting, and heterogeneity-aware optimization can transform static pipelines into self-optimizing systems, while also identifying future research directions in reinforcement learning–based orchestration, multi-cloud cost modeling, and explainable AI for cloud execution environments.

Zitieren

APA

Seetala, S. R. (2023). Intelligent Orchestration : AI-Driven Optimization of Cloud Data Pipelines in Heterogeneous and Serverless Environments. International Journal of Scientific Research in Science Engineering and Technology, 876. https://doi.org/10.32628/ijsrset23107111

Vancouver

Seetala SR. Intelligent Orchestration : AI-Driven Optimization of Cloud Data Pipelines in Heterogeneous and Serverless Environments. International Journal of Scientific Research in Science Engineering and Technology. 2023:876. doi:10.32628/ijsrset23107111

BibTeX
@article{Seetala2023,
  author = {Seetala, Srinivasa Rao},
  title = {Intelligent Orchestration : AI-Driven Optimization of Cloud Data Pipelines in Heterogeneous and Serverless Environments},
  journal = {International Journal of Scientific Research in Science Engineering and Technology},
  year = {2023},
  pages = {876},
  doi = {10.32628/ijsrset23107111}
}
Kostenlos starten →