EUVIMEDEUVIMED entdecken
Crossref

Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams

Aditya Rautaray, T. Shynu, R. Steffi, B. Sudha

AVE Trends in Intelligent Computing Systems · 2026

DOI: 10.64091/atics.2026.000259 ↗

Kurz gefasst

Fachartikel in der Zeitschrift AVE Trends in Intelligent Computing Systems, 2026. Verfasst von Aditya Rautaray und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper examines the problem as a specific instance of schema drift in high-velocity data streams and proposes a metadata-driven, adaptive ETL (Extract-Transform-Load) framework to address it. Traditional Data Integration tools come with these soft, brittle monitors, which fail down even if an upstream source changes its structure without prior notification. This weakness results in significant data loss and downtime. The architecture system uses event streaming (Apache Kafka), distributed processing (Apache Spark), and a dynamic Schema Registry abstraction to eliminate the need for a payload structure definition. The simulation has been performed on a synthetic dataset of 410 distinct IoT sensor telemetry cases, intentionally designed to simulate high-frequency changes in field types, column additions, and changes in granularity. A comparison with a typical static schema validation approach shows that the adaptive architecture yields substantially greater system robustness. The experiments reveal that the metadata-driven strategy achieves 99% system uptime during drift events, compared with 0% for static systems. The Results indicate that by changing the validation logic from hard-coded scripts to a dynamic metadata layer, organisations can build agile immunity against volatility in real-time data streams.

Zitieren

APA

Rautaray, A., Shynu, T., Steffi, R., & Sudha, B. (2026). Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams. AVE Trends in Intelligent Computing Systems. https://doi.org/10.64091/atics.2026.000259

Vancouver

Rautaray A, Shynu T, Steffi R, Sudha B. Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams. AVE Trends in Intelligent Computing Systems. 2026. doi:10.64091/atics.2026.000259

BibTeX
@article{Rautaray2026,
  author = {Rautaray, Aditya and Shynu, T. and Steffi, R. and Sudha, B.},
  title = {Adaptive Metadata Driven Extract-Transform-Load for Schema Drift Immunity in High Velocity Event Streams},
  journal = {AVE Trends in Intelligent Computing Systems},
  year = {2026},
  doi = {10.64091/atics.2026.000259}
}
Kostenlos starten →