EUVIMEDEUVIMED entdecken
Crossref

Scalable Data Engineering Pipelines for Real-Time Analytics in Big Data Environments

Reddy Srikanth Madhuranthakam

FMDB Transactions on Sustainable Computing Systems · 2024

DOI: 10.69888/ftscs.2024.000262 ↗

Kurz gefasst

Fachartikel in der Zeitschrift FMDB Transactions on Sustainable Computing Systems, 2024. Verfasst von Reddy Srikanth Madhuranthakam.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the world becoming increasingly data-driven, actionable insights and decisions that depend on real-time analytics have been at the forefront. However, processing huge volumes of data in real time requires very strong, scalable, and efficient data engineering pipelines. This paper describes the design, development, and optimization of scalable data engineering pipelines for real-time analytics in big data environments. Ingestion, processing, storage and visualization, along with their interactions within the distributed computing setup, will all be part of the paper. More best practices will be presented regarding handling high-velocity data streams to become fault-tolerant and data consistency. We design a system architecture using the empirical approach: bringing real-time data processing frameworks like Apache Kafka and Apache Flink with the best available cloud-based storage solution to achieve the scale-out seamlessness of the data processing task. Alongside the design comes the performance evaluation results for a comparison among varied strategies to be followed while carrying out real-time analytics focusing on scalability, throughput, and latency. It indicates that the efficiency of the proposed pipeline is much improved in processing and is well-suited for large-scale, real-time analytics for numerous industries.

Zitieren

APA

Madhuranthakam, R. S. (2024). Scalable Data Engineering Pipelines for Real-Time Analytics in Big Data Environments. FMDB Transactions on Sustainable Computing Systems. https://doi.org/10.69888/ftscs.2024.000262

Vancouver

Madhuranthakam RS. Scalable Data Engineering Pipelines for Real-Time Analytics in Big Data Environments. FMDB Transactions on Sustainable Computing Systems. 2024. doi:10.69888/ftscs.2024.000262

BibTeX
@article{Madhuranthakam2024,
  author = {Madhuranthakam, Reddy Srikanth},
  title = {Scalable Data Engineering Pipelines for Real-Time Analytics in Big Data Environments},
  journal = {FMDB Transactions on Sustainable Computing Systems},
  year = {2024},
  doi = {10.69888/ftscs.2024.000262}
}
Kostenlos starten →