Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses
Proceedings of the VLDB Endowment · 2025-08 · 18 (12) · 4887-4895
DOI: 10.14778/3750601.3750613 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 16.09.2025. Verfasst von Hongtao Yang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 19 Literaturangaben.
Abstract (Originalsprache)
Ensuring the reliability of data pipelines is critical for modern data-driven organizations, yet building robust Continuous Integration (CI) in large, distributed data warehouses remains a significant challenge. Complexities arising from distributed ownership, the high cost of replicating production environments, and the rapid evolution of business logic lead to fragile pipelines and costly failures. This paper introduces a novel CI framework designed to conquer these challenges, achieving 94.5% pre-production issue detection in YouTube's data warehouse while dramatically reducing resource consumption. Our key innovation lies in a production-coniguration-driven testing methodology, that enables scalable, isolated testing directly within the production environment. This approach reduces testing overhead and ensures high test fidelity. Furthermore, we present a lineage-aware impact analysis framework that automatically propagates data quality checks across distributed pipeline components based on an algebraic dependency model, ensuring data consistency and promoting cross-team collaboration. This production-proven solution provides a practical blueprint for CI/CD in complex, large-scale environments.
Zitieren
APA
Yang, H., Xu, Z., Yudin, S., & Davidson, A. (2025). Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses. Proceedings of the VLDB Endowment, 18(12), 4887-4895. https://doi.org/10.14778/3750601.3750613
Vancouver
Yang H, Xu Z, Yudin S, Davidson A. Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses. Proceedings of the VLDB Endowment. 2025;18(12):4887-4895. doi:10.14778/3750601.3750613
BibTeX
@article{Yang2025,
author = {Yang, Hongtao and Xu, Zhichen and Yudin, Sergey and Davidson, Andrew},
title = {Unlocking the Power of CI/CD for Data Pipelines in Distributed Data Warehouses},
journal = {Proceedings of the VLDB Endowment},
year = {2025},
volume = {18},
number = {12},
pages = {4887--4895},
doi = {10.14778/3750601.3750613}
}
EUVIMED