AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions
Journal of Engineering Research and Sciences · 2026-03 · 5 (3) · 1-13
DOI: 10.55708/js0503001 ↗At a glance
Journal article in Journal of Engineering Research and Sciences, 2026-03. Written by Sowjanya Deva and 1 co-authors. 36 references.
Abstract (original language)
Cloud data lakes require continuous optimization across multiple dimensions: physical design (partitioning, compression), query execution, and data quality assurance. This paper presents AIDALOS (AI-Driven Autonomous Data Lake Optimization System), a framework that integrates quality monitoring with physical optimization decisions. The system uses reinforcement learning to adapt monitoring intensity and trigger physical design changes based on detected anomalies, drift patterns, and workload shifts. Deep Q-networks learn when to repartition tables, ensemble models select compression codecs based on data characteristics and access patterns, and neural cost estimators improve query plan selection. Our evaluation across five machine learning pipelines demonstrates that this integrated approach achieves 47% storage cost reduction and 62% query performance improvement compared to static configurations, with 89.9% F1-score for quality issue detection. The key insight is that quality signals drift detection, anomaly patterns, and workload changes should directly inform physical optimization decisions rather than treating these as separate concerns.
Cite
APA
Deva, S., & Chintacunta, S. N. R. (2026). AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions. Journal of Engineering Research and Sciences, 5(3), 1-13. https://doi.org/10.55708/js0503001
Vancouver
Deva S, Chintacunta SNR. AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions. Journal of Engineering Research and Sciences. 2026;5(3):1-13. doi:10.55708/js0503001
BibTeX
@article{Deva2026,
author = {Deva, Sowjanya and Chintacunta, Surya Narayana Reddy},
title = {AI-Driven Data Lake Optimization: Integrating Quality Monitoring with Intelligent Physical Design Decisions},
journal = {Journal of Engineering Research and Sciences},
year = {2026},
volume = {5},
number = {3},
pages = {1--13},
doi = {10.55708/js0503001}
}
EUVIMED