EUVIMEDEUVIMED entdecken
Crossref

Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses

Hongtao Yang, Zhichen Xu, Sergey Yudin, Andrew Davidson

Proceedings of the VLDB Endowment · 2025-08 · 18 (12) · 4887-4895

DOI: 10.14778/3750601.3750613 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 16.09.2025. Verfasst von Hongtao Yang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 19 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Ensuring the reliability of data pipelines is critical for modern data-driven organizations, yet building robust Continuous Integration (CI) in large, distributed data warehouses remains a significant challenge. Complexities arising from distributed ownership, the high cost of replicating production environments, and the rapid evolution of business logic lead to fragile pipelines and costly failures. This paper introduces a novel CI framework designed to conquer these challenges, achieving 94.5% pre-production issue detection in YouTube's data warehouse while dramatically reducing resource consumption. Our key innovation lies in a production-coniguration-driven testing methodology, that enables scalable, isolated testing directly within the production environment. This approach reduces testing overhead and ensures high test fidelity. Furthermore, we present a lineage-aware impact analysis framework that automatically propagates data quality checks across distributed pipeline components based on an algebraic dependency model, ensuring data consistency and promoting cross-team collaboration. This production-proven solution provides a practical blueprint for CI/CD in complex, large-scale environments.

Zitieren

APA

Yang, H., Xu, Z., Yudin, S., & Davidson, A. (2025). Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses. Proceedings of the VLDB Endowment, 18(12), 4887-4895. https://doi.org/10.14778/3750601.3750613

Vancouver

Yang H, Xu Z, Yudin S, Davidson A. Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses. Proceedings of the VLDB Endowment. 2025;18(12):4887-4895. doi:10.14778/3750601.3750613

BibTeX
@article{Yang2025,
  author = {Yang, Hongtao and Xu, Zhichen and Yudin, Sergey and Davidson, Andrew},
  title = {Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses},
  journal = {Proceedings of the VLDB Endowment},
  year = {2025},
  volume = {18},
  number = {12},
  pages = {4887--4895},
  doi = {10.14778/3750601.3750613}
}
Kostenlos starten →