A Reproducible Benchmark-Validity Audit and Calibration Study for Cross-Home Fault Diagnosis in Smart-Home Sensor Systems
Sensors · 07.08.2026 · 26 (16) · 5025
DOI: 10.3390/s26165025 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 07.08.2026. Verfasst von Norkobil Saydirasulovich Saydirasulov und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Engineering, Gachon University, Seongnam-si 13120, Republic of Korea. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, MOE, Ministry of Education.
Abstract (Originalsprache)
Diagnosing faults across different smart homes is hard: sensor names, layouts, and daily routines differ from home to home, so a model trained in one home rarely works in another. We study an ontology-guided framework for cross-home fault diagnosis, but our main contribution is a benchmark-validity audit—a systematic check of whether the datasets used to evaluate such systems actually measure fault detection. Using the public Center for Advanced Studies in Adaptive Systems (CASAS) smart-home datasets (homes hh101–hh110) and real household power data (HomeC, UMass Smart*), we show that much of the high cross-home accuracy reported on these benchmarks is an artifact of features that re-encode the labelling rules rather than evidence of transfer: when those features are removed, the macro-averaged F1 score (macro-F1) collapses toward the level obtained with randomly permuted labels. We therefore treat these datasets as semantic-transfer and benchmark-validity studies, not fault-detection results. The framework’s distinguishing component is a counterfactual calibration layer that returns a probability for its recommended intervention; on a controlled structural causal model with known interventions, it achieves a Brier skill score of 0.369 for intervention-success probabilities. Separately, on the simulation-derived LBNL Fan Coil Unit benchmark, a conventional gradient-boosted multiclass fault classifier achieves accuracy comparable to a random forest but about six times lower expected calibration error (0.026 vs. 0.159) under a scenario-matched split. This calibration advantage does not generalize to held-out simulation scenarios, where the calibration error rises to 0.372; we report this negative result as a limitation. We are explicit about scope: the ontology reasoner and the real-stream causal graph are only partially implemented, and the counterfactual recommendations are validated only under controlled or simulated conditions, not in deployed homes. The results are intended for researchers who build or benchmark sensor-based fault-diagnosis models, for dataset curators, and for practitioners who need calibrated rather than merely accurate outputs. All code, the proxy-label rules, and the leakage audit are released.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Engineering, Gachon University, Seongnam-si 13120, Republic of Korea.
- Department of Engineering and Communications, ZARMED University, Samarkand Campus, Khodja Akhror Vali Street 23, Samarkand 140100, Uzbekistan.
- Department of Artificial Intelligence, Samarkand State University, Samarkand 140104, Uzbekistan.
- Department of Computer Engineering, Korea National University of Transportation, Chungju-si 27469, Republic of Korea.
- Department of Construction Engineering, Samarkand State Technical University, Samarkand 140100, Uzbekistan.
- Department of Information Technology, Samarkand Institute of Economics and Service, Samarkand 140100, Uzbekistan.
ORCID-Profile: Norkobil Saydirasulovich Saydirasulov, Abror Shavkatovich Buriboev, Shuxrat Isroilov, Cheolwon Lee
Förderung
- National Research Foundation of Korea
- MOE
- Ministry of Education
Zitieren
APA
Saydirasulov, N. S., Buriboev, A. S., Isroilov, S., Oh, R., Buribayev, S., Abduvaytov, A., Umirov, J., Badalov, J. I., Axmedova, A., Lee, C., & Jeon, H. S. (2026). A Reproducible Benchmark-Validity Audit and Calibration Study for Cross-Home Fault Diagnosis in Smart-Home Sensor Systems. Sensors, 26(16), 5025. https://doi.org/10.3390/s26165025
Vancouver
Saydirasulov NS, Buriboev AS, Isroilov S, Oh R, Buribayev S, Abduvaytov A, et al. A Reproducible Benchmark-Validity Audit and Calibration Study for Cross-Home Fault Diagnosis in Smart-Home Sensor Systems. Sensors. 2026;26(16):5025. doi:10.3390/s26165025
BibTeX
@article{Saydirasulov2026,
author = {Saydirasulov, Norkobil Saydirasulovich and Buriboev, Abror Shavkatovich and Isroilov, Shuxrat and Oh, Ryumduck and Buribayev, Shavkat and Abduvaytov, Abbos and Umirov, Jamshid and Badalov, Jasur Ismailovich and Axmedova, Aziza and Lee, Cheolwon and Jeon, Heung Seok},
title = {A Reproducible Benchmark-Validity Audit and Calibration Study for Cross-Home Fault Diagnosis in Smart-Home Sensor Systems},
journal = {Sensors},
year = {2026},
volume = {26},
number = {16},
pages = {5025},
doi = {10.3390/s26165025}
}
EUVIMED