EUVIMEDOdkrijte EUVIMED
Crossref

AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions

Sowjanya Deva, Surya Narayana Reddy Chintacunta

Journal of Engineering Research and Sciences · 2026-03 · 5 (3) · 1-13

DOI: 10.55708/js0503001 ↗

At a glance

Journal article in Journal of Engineering Research and Sciences, 2026-03. Written by Sowjanya Deva and 1 co-authors. 36 references.

AbstraktBibliografické údajeCiteRelated studies

Abstract (original language)

Cloud data lakes require continuous optimization across multiple dimensions: physical design (partitioning, compression), query execution, and data quality assurance. This paper presents AIDALOS (AI-Driven Autonomous Data Lake Optimization System), a framework that integrates quality monitoring with physical optimization decisions. The system uses reinforcement learning to adapt monitoring intensity and trigger physical design changes based on detected anomalies, drift patterns, and workload shifts. Deep Q-networks learn when to repartition tables, ensemble models select compression codecs based on data characteristics and access patterns, and neural cost estimators improve query plan selection. Our evaluation across five machine learning pipelines demonstrates that this integrated approach achieves 47% storage cost reduction and 62% query performance improvement compared to static configurations, with 89.9% F1-score for quality issue detection. The key insight is that quality signals drift detection, anomaly patterns, and workload changes should directly inform physical optimization decisions rather than treating these as separate concerns.

Cite

APA

Deva, S., & Chintacunta, S. N. R. (2026). AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions. Journal of Engineering Research and Sciences, 5(3), 1-13. https://doi.org/10.55708/js0503001

Vancouver

Deva S, Chintacunta SNR. AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions. Journal of Engineering Research and Sciences. 2026;5(3):1-13. doi:10.55708/js0503001

BibTeX
@article{Deva2026,
  author = {Deva, Sowjanya and Chintacunta, Surya Narayana Reddy},
  title = {AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions},
  journal = {Journal of Engineering Research and Sciences},
  year = {2026},
  volume = {5},
  number = {3},
  pages = {1--13},
  doi = {10.55708/js0503001}
}
Start for free →