Enhancing Data Engineering Workflows Using Generative AI Agents
Advances in Nonlinear Variational Inequalities · 26.03.2025 · 28 (7s) · 1448-1457
DOI: 10.52783/anvi.v28.7069 ↗Kurz gefasst
Fachartikel in der Zeitschrift Advances in Nonlinear Variational Inequalities, online erschienen am 26.03.2025. Verfasst von Jay Kumar Kushwaha.
Abstract (Originalsprache)
Modern data engineering teams face increasing challenges as data comes from diverse sources such as relational databases, streaming platforms, APIs, and cloud data lakes. Although orchestration tools like Apache Airflow and Prefect have improved pipeline management, building and maintaining ETL workflows still requires significant expertise, time, and effort. This paper introduces GenETL, a multi-agent Generative AI framework that helps automate key data engineering tasks, including schema discovery, pipeline generation, anomaly detection, data lineage tracking, and self-correcting execution. The framework uses specialized AI agents—such as a Schema Analyst, Pipeline Architect, Quality Sentinel, and Lineage Recorder—that collaborate under a central orchestrator powered by a fine-tuned large language model (LLM). Experiments on five enterprise datasets showed strong results, achieving up to 98.2% pipeline accuracy, 94.7% schema evolution handling, and 91.3% multi-source join correctness, while reducing overall pipeline development time by an average of 87%. This work demonstrates how Generative AI can make ETL development faster, more reliable, and easier to manage in modern data engineering environments.
Zitieren
APA
Kushwaha, J. K. (2025). Enhancing Data Engineering Workflows Using Generative AI Agents. Advances in Nonlinear Variational Inequalities, 28(7s), 1448-1457. https://doi.org/10.52783/anvi.v28.7069
Vancouver
Kushwaha JK. Enhancing Data Engineering Workflows Using Generative AI Agents. Advances in Nonlinear Variational Inequalities. 2025;28(7s):1448-1457. doi:10.52783/anvi.v28.7069
BibTeX
@article{Kushwaha2025,
author = {Kushwaha, Jay Kumar},
title = {Enhancing Data Engineering Workflows Using Generative AI Agents},
journal = {Advances in Nonlinear Variational Inequalities},
year = {2025},
volume = {28},
number = {7s},
pages = {1448--1457},
doi = {10.52783/anvi.v28.7069}
}
EUVIMED