Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams
AVE Trends in Intelligent Computing Systems · 2026
DOI: 10.64091/atics.2026.000259 ↗Kurz gefasst
Fachartikel in der Zeitschrift AVE Trends in Intelligent Computing Systems, 2026. Verfasst von Aditya Rautaray und 3 weiteren Autor:innen.
Abstract (Originalsprache)
This paper examines the problem as a specific instance of schema drift in high-velocity data streams and proposes a metadata-driven, adaptive ETL (Extract-Transform-Load) framework to address it. Traditional Data Integration tools come with these soft, brittle monitors, which fail down even if an upstream source changes its structure without prior notification. This weakness results in significant data loss and downtime. The architecture system uses event streaming (Apache Kafka), distributed processing (Apache Spark), and a dynamic Schema Registry abstraction to eliminate the need for a payload structure definition. The simulation has been performed on a synthetic dataset of 410 distinct IoT sensor telemetry cases, intentionally designed to simulate high-frequency changes in field types, column additions, and changes in granularity. A comparison with a typical static schema validation approach shows that the adaptive architecture yields substantially greater system robustness. The experiments reveal that the metadata-driven strategy achieves 99% system uptime during drift events, compared with 0% for static systems. The Results indicate that by changing the validation logic from hard-coded scripts to a dynamic metadata layer, organisations can build agile immunity against volatility in real-time data streams.
Zitieren
APA
Rautaray, A., Shynu, T., Steffi, R., & Sudha, B. (2026). Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams. AVE Trends in Intelligent Computing Systems. https://doi.org/10.64091/atics.2026.000259
Vancouver
Rautaray A, Shynu T, Steffi R, Sudha B. Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams. AVE Trends in Intelligent Computing Systems. 2026. doi:10.64091/atics.2026.000259
BibTeX
@article{Rautaray2026,
author = {Rautaray, Aditya and Shynu, T. and Steffi, R. and Sudha, B.},
title = {Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams},
journal = {AVE Trends in Intelligent Computing Systems},
year = {2026},
doi = {10.64091/atics.2026.000259}
}
EUVIMED