Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task
We introduce a challenging decision-making task that we call active acquisition for multimodal temporal data (A2MT). In many real-world scenarios, input features are not readily available at test time and must instead be acquired at significant cost. With A2MT, we aim to learn agents that actively select which modalities of an input to acquire, trading off acquisition cost and predictive performance. A2MT extends a previous task called active feature acquisition to temporal decision making about high-dimensional inputs. We propose a method based on the Perceiver IO architecture to address A2MT in practice. Our agents are able to solve a novel synthetic scenario requiring practically relevant cross-modal reasoning skills. On two large-scale, real-world datasets, Kinetics-700 and AudioSet, our agents successfully learn cost-reactive acquisition behavior. However, an ablation reveals they are unable to learn adaptive acquisition strategies, emphasizing the difficulty of the task even for state-of-the-art models. Applications of A2MT may be impactful in domains like medicine, robotics, or finance, where modalities differ in acquisition cost and informativeness.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingInformativenessMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MAViC: Multimodal Active Learning for Video Captioning
A large number of annotated video-caption pairs are required for training video captioning models, resulting in high annotation costs. Active learning can be instrumental in reducing these annotation requirements. Howeve…
Active LearningDecoderSemantic SimilaritySemantic Textual Similarity+2Relaxed Efficient Acquisition of Context and Temporal Features
In many biomedical applications, measurements are not freely available at inference time: each laboratory test, imaging modality, or assessment incurs financial cost, time burden, or patient risk. Longitudinal active fea…
A Mutual Information Lower Bound for Multimodal Regression Active Learning
Active learning for continuous regression has lacked an acquisition function that targets epistemic uncertainty when the predictive distribution is multimodal: variance misses modal disagreement, and information-theoreti…
Active LearningSense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence
Edge-based multimodal medical monitoring requires models that balance diagnostic accuracy with severe energy constraints. Continuous acquisition of ECG, PPG, EMG, and IMU streams rapidly drains wearable batteries, often …
GBU-Palm: A Multimodal Video Dataset and Benchmark for Palm Presentation Attack Detection
Existing palm presentation attack detection (PAD) datasets are often limited by static imagery, restricted acquisition conditions, or insufficient multimodal video data, hindering systematic evaluation across environment…