EUVIMEDEUVIMED entdecken
Crossref

Enhancing Data Engineering Workflows Using Generative AI Agents

Jay Kumar Kushwaha

Advances in Nonlinear Variational Inequalities · 26.03.2025 · 28 (7s) · 1448-1457

DOI: 10.52783/anvi.v28.7069 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Advances in Nonlinear Variational Inequalities, online erschienen am 26.03.2025. Verfasst von Jay Kumar Kushwaha.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Modern data engineering teams face increasing challenges as data comes from diverse sources such as relational databases, streaming platforms, APIs, and cloud data lakes. Although orchestration tools like Apache Airflow and Prefect have improved pipeline management, building and maintaining ETL workflows still requires significant expertise, time, and effort. This paper introduces GenETL, a multi-agent Generative AI framework that helps automate key data engineering tasks, including schema discovery, pipeline generation, anomaly detection, data lineage tracking, and self-correcting execution. The framework uses specialized AI agents—such as a Schema Analyst, Pipeline Architect, Quality Sentinel, and Lineage Recorder—that collaborate under a central orchestrator powered by a fine-tuned large language model (LLM). Experiments on five enterprise datasets showed strong results, achieving up to 98.2% pipeline accuracy, 94.7% schema evolution handling, and 91.3% multi-source join correctness, while reducing overall pipeline development time by an average of 87%. This work demonstrates how Generative AI can make ETL development faster, more reliable, and easier to manage in modern data engineering environments.

Zitieren

APA

Kushwaha, J. K. (2025). Enhancing Data Engineering Workflows Using Generative AI Agents. Advances in Nonlinear Variational Inequalities, 28(7s), 1448-1457. https://doi.org/10.52783/anvi.v28.7069

Vancouver

Kushwaha JK. Enhancing Data Engineering Workflows Using Generative AI Agents. Advances in Nonlinear Variational Inequalities. 2025;28(7s):1448-1457. doi:10.52783/anvi.v28.7069

BibTeX
@article{Kushwaha2025,
  author = {Kushwaha, Jay Kumar},
  title = {Enhancing Data Engineering Workflows Using Generative AI Agents},
  journal = {Advances in Nonlinear Variational Inequalities},
  year = {2025},
  volume = {28},
  number = {7s},
  pages = {1448--1457},
  doi = {10.52783/anvi.v28.7069}
}
Kostenlos starten →