AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing
INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · 25.03.2026 · 10 (03) · 1-9
DOI: 10.55041/ijsrem58245 ↗Kurz gefasst
Fachartikel in der Zeitschrift INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, online erschienen am 25.03.2026. Verfasst von Girish Dashmukhe und 4 weiteren Autor:innen.
Abstract (Originalsprache)
Modern enterprises ingest vast amounts of data from diverse sources, requiring robust ETL (Extract-Transform-Load) pipelines. Traditional pipelines often break due to data inconsistencies or infrastructure faults. This paper proposes a self-healing ETL pipeline architecture built on AWS serverless services (S3, Glue, Lambda, Step Functions, etc.) augmented with AI/ML for automatic error detection and recovery. In our design, raw data lands in Amazon S3, triggers AWS Glue jobs for transformation, and any failures are detected via anomaly detection (e.g. using AWS Glue Data Quality or a SageMaker-trained model). A Step Functions workflow coordinates the ETL tasks and can retry or invoke corrective routines when errors occur. We include the architecture diagram, workflow, methodology, and cost analysis. Experimental results from related work show that self-healing pipelines significantly improve uptime and reduce manual intervention. KEYWORDS Cloud Computing, AWS, ETL Pipeline, Serverless Architecture, Artificial Intelligence, Machine Learning, Data Transformation, Data Quality, Automation, Data Pipeline.
Zitieren
APA
Dashmukhe, G., Bhat, Y. V., Badhe, M., Khande, V., & Patel, V. (2026). AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing. INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, 10(03), 1-9. https://doi.org/10.55041/ijsrem58245
Vancouver
Dashmukhe G, Bhat YV, Badhe M, Khande V, Patel V. AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing. INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT. 2026;10(03):1-9. doi:10.55041/ijsrem58245
BibTeX
@article{Dashmukhe2026,
author = {Dashmukhe, Girish and Bhat, Yash Vinod and Badhe, Manasi and Khande, Vaishnavi and Patel, Vishakha},
title = {AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing},
journal = {INTERNATIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT},
year = {2026},
volume = {10},
number = {03},
pages = {1--9},
doi = {10.55041/ijsrem58245}
}
EUVIMED