SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor Interactions
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · 03.09.2025 · 9 (3) · 1-35
DOI: 10.1145/3749496 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, online erschienen am 03.09.2025. Verfasst von Xiaofan Yu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: National Science Foundation, Semiconductor Research Corporation.
Abstract (Originalsprache)
Natural language interaction with sensing systems is crucial for addressing users' personal concerns and providing health-related insights into their daily lives. When a user asks a question, the system automatically analyzes the full history of sensor data, extracts relevant information, and generates an appropriate response. However, existing systems are limited to short-duration (e.g., one minute) or low-frequency (e.g., daily step count) sensor data. In addition, they struggle with quantitative questions that require precise numerical answers. In this work, we introduce SensorChat, the first end-to-end QA system designed for daily life monitoring using long-duration, high-frequency time series data. Given raw sensor signals spanning multiple days and a user-defined natural language question, SensorChat generates semantically meaningful responses that directly address users' concerns. SensorChat effectively handles both quantitative questions that require numerical precision and qualitative questions that require high-level reasoning to infer subjective insights. To achieve this, SensorChat uses an innovative three-stage pipeline that includes question decomposition, sensor data query, and answer assembly. The first and third stages leverage Large Language Models (LLMs) to interpret human queries and generate responses. The intermediate querying stage extracts relevant information from the complete sensor data history, which is then combined with the original query in the final stage to produce accurate and meaningful answers. Real-world implementations demonstrate SensorChat's capability for real-time interactions on a cloud server while also being able to run entirely on an edge platform after quantization. Comprehensive QA evaluations show that SensorChat achieves 93% higher answer accuracy than the best performing state-of-the-art systems on quantitative questions. Furthermore, a user study with eight volunteers highlights SensorChat's effectiveness in answering qualitative and open-ended questions. The code is available at https://github.com/Orienfish/SensorChat.
Förderung
- National Science Foundation
- Semiconductor Research Corporation
Zitieren
APA
Yu, X., Hu, L., Reichman, B., Chu, D., Chandrupatla, R., Zhang, X., Heck, L., & Rosing, T. S. (2025). SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(3), 1-35. https://doi.org/10.1145/3749496
Vancouver
Yu X, Hu L, Reichman B, Chu D, Chandrupatla R, Zhang X, et al. SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2025;9(3):1-35. doi:10.1145/3749496
BibTeX
@article{Yu2025,
author = {Yu, Xiaofan and Hu, Lanxiang and Reichman, Benjamin and Chu, Dylan and Chandrupatla, Rushil and Zhang, Xiyuan and Heck, Larry and Rosing, Tajana S.},
title = {SensorChat: Answering Qualitative and Quantitative Questions during Long-term Multimodal Sensor Interactions},
journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
year = {2025},
volume = {9},
number = {3},
pages = {1--35},
doi = {10.1145/3749496}
}
EUVIMED