SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions
Natural language interaction with sensing systems is crucial for addressing users' personal concerns and providing health-related insights into their daily lives. When a user asks a question, the system automatically analyzes the full history of sensor data, extracts relevant information, and generates an appropriate response. However, existing systems are limited to short-duration (e.g., one minute) or low-frequency (e.g., daily step count) sensor data. In addition, they struggle with quantitative questions that require precise numerical answers. In this work, we introduce SensorChat, the first end-to-end QA system designed for daily life monitoring using long-duration, high-frequency time series data. Given raw sensor signals spanning multiple days and a user-defined natural language question, SensorChat generates semantically meaningful responses that directly address user concerns. SensorChat effectively handles both quantitative questions that require numerical precision and qualitative questions that require high-level reasoning to infer subjective insights. To achieve this, SensorChat uses an innovative three-stage pipeline including question decomposition, sensor data query, and answer assembly. The first and third stages leverage Large Language Models (LLMs) to interpret human queries and generate responses. The intermediate querying stage extracts relevant information from the complete sensor data history. Real-world implementation demonstrate SensorChat's capability for real-time interactions on a cloud server while also being able to run entirely on edge platforms after quantization. Comprehensive QA evaluations show that SensorChat achieves up to 93% higher answer accuracy than state-of-the-art systems on quantitative questions. Additionally, a user study with eight volunteers highlights SensorChat's effectiveness in answering qualitative and open-ended questions.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationQuestion AnsweringSimilar Papers 제목 키워드 기반
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
Many natural language questions require qualitative, quantitative or logical comparisons between two entities or events. This paper addresses the problem of improving the accuracy and consistency of responses to comparis…
Data AugmentationMachine Reading ComprehensionMultiple-choiceQuestion Answering+1Bypassing Document Ingestion: An MCP Approach to Financial Q&A
Answering financial questions is often treated as an information retrieval problem. In practice, however, much of the relevant information is already available in curated vendor systems, especially for quantitative analy…
Information RetrievalQuestion AnsweringAsking and Answering Questions to Extract Event-Argument Structures
This paper presents a question-answering approach to extract document-level event-argument structures. We automatically ask and answer questions for each argument type an event may have. Questions are generated using man…
coreference-resolutionCoreference ResolutionData AugmentationQuestion Answering+1Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)