A Hidden Markov Framework for Concept Drift Detection and Classification
Journal of Visual Artificial Intelligence · 24.06.2026 · 1 (1)
DOI: 10.64189/vai.26102 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Visual Artificial Intelligence, online erschienen am 24.06.2026. Verfasst von Kailas Kunjumon und 4 weiteren Autor:innen.
Abstract (Originalsprache)
Machine learning systems deployed in real-world environments frequently encounter non-stationary data streams in which the underlying data-generating distribution shifts over time. This phenomenon, known as concept drift, causes progressive model degradation if left undetected. Existing detection methods largely treat drift as a binary event, ignoring the temporal dynamics and structural diversity of distributional change. In this paper, we present the Hidden Markov Model (HMM)-based drift tracking (HDT) system, a framework that models concept drift as a latent probabilistic process over three hidden states, i.e., stable, warning, and drift using an HMM. The Viterbi algorithm is employed to decode the most probable state sequence from a multivariate observation vector constructed from sliding-window statistical features of the data stream, including the sample mean, variance, Kolmogorov-Smirnov(KS) statistic, and model error rate. Upon detecting a drift event, HDT classifies it into one of two primary structural categories as sudden drift and gradual drift and further determines whether each event is harmful or benign based on a feature-derived severity criterion. Experiments conducted on the university of California (UCI) gas sensor array drift dataset, comprising 13,810 post-initialization observations across ten sensor batches collected over 36 months, demonstrate the system's ability to track drift onset, progression, and recovery in a physically motivated non-stationary stream. Results shows 2,575 confirmed drift events, with 1,021 classified as harmful and 1,554 as non-harmful. The HDT system offers a principled and interpretable alternative to threshold-based detectors for monitoring deployed machine learning systems in dynamic environments.
Zitieren
APA
Kunjumon, K., Paul, A. M., Joby, M. K., KS, H., & Misra, B. (2026). A Hidden Markov Framework for Concept Drift Detection and Classification. Journal of Visual Artificial Intelligence, 1(1). https://doi.org/10.64189/vai.26102
Vancouver
Kunjumon K, Paul AM, Joby MK, KS H, Misra B. A Hidden Markov Framework for Concept Drift Detection and Classification. Journal of Visual Artificial Intelligence. 2026;1(1). doi:10.64189/vai.26102
BibTeX
@article{Kunjumon2026,
author = {Kunjumon, Kailas and Paul, Aksa Maria and Joby, Mariya K and KS, Hemanth and Misra, Bitan},
title = {A Hidden Markov Framework for Concept Drift Detection and Classification},
journal = {Journal of Visual Artificial Intelligence},
year = {2026},
volume = {1},
number = {1},
doi = {10.64189/vai.26102}
}
EUVIMED