Task-Oriented Mulsemedia Communication using Unified Perceiver and Conformal Prediction in 6G Wireless Systems
The growing prominence of eXtended Reality (XR), holographic-type communications, and metaverse demands truly immersive user experiences by using many sensory modalities, including sight, hearing, touch, smell, taste, etc. Additionally, the widespread deployment of sensors in areas such as agriculture, manufacturing, and smart homes is generating diverse sensory data. A new media format known as multisensory media (mulsemedia) has emerged, which incorporates many sensory modalities beyond the traditional visual and auditory media. 6G wireless systems are envisioned to support the Internet of Senses, making it crucial to explore effective data fusion and communication strategies for mulsemedia. In this paper, we introduce a task-oriented multi-task mulsemedia communication system named MuSeCo, which is developed using unified Perceiver models and Conformal Prediction. This unified model can accept any sensory input and efficiently extract latent semantic features, making it adaptable for deployment across various Artificial Intelligence of Things (AIoT) devices. Conformal Prediction is employed for modality selection and combination, enhancing task accuracy while minimizing data communication overhead. The model is trained using six sensory modalities across four classification tasks. Simulations and experiments demonstrate that it can effectively fuse sensory modalities, significantly reduce end-to-end communication latency and energy consumption, and maintain high accuracy in communication-constrained systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Conformal PredictionSimilar Papers 제목 키워드 기반
Can Multisensory Cues in VR Help Train Pattern Recognition to Citizen Scientists?
As the internet of things (IoT) has integrated physical and digital technologies, designing for multiple sensory media (mulsemedia) has become more attainable. Designing technology for multiple senses has the capacity to…
Transfer LearningAdaPerceiver: Transformers with Adaptive Width, Depth, and Tokens
Modern transformer architectures achieve remarkable performance across tasks and domains but remain rigid in how they allocate computation at inference time. Real-world deployment often requires models to adapt to divers…
Semantic SegmentationImage ClassificationDepth EstimationUni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
Despite the remarkable success of foundation models, their task-specific fine-tuning paradigm makes them inconsistent with the goal of general perception modeling. The key to eliminating this inconsistency is to use gene…
DecoderLanguage ModellingMulti-Task LearningUni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
Biological intelligence systems of animals perceive the world by integrating information in different modalities and processing simultaneously for various tasks. In contrast, current machine learning research follows a t…
A Unified Multi-Task Semantic Communication System for Multimodal Data
Task-oriented semantic communications have achieved significant performance gains. However, the employed deep neural networks in semantic communications have to be updated when the task is changed or multiple models need…
feature selectionSemantic Communication