Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning
This work focuses on learning useful and robust deep world models using multiple, possibly unreliable, sensors. We find that current methods do not sufficiently encourage a shared representation between modalities; this can cause poor performance on downstream tasks and over-reliance on specific sensors. As a solution, we contribute a new multi-modal deep latent state-space model, trained using a mutual information lower-bound. The key innovation is a specially-designed density ratio estimator that encourages consistency between the latent codes of each modality. We tasked our method to learn policies (in a self-supervised manner) on multi-modal Natural MuJoCo benchmarks and a challenging Table Wiping task. Experiments show our method significantly outperforms state-of-the-art deep reinforcement learning methods, particularly in the presence of missing observations.
Code (1)
Tasks
Deep Reinforcement LearningMuJoCoreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Machine Learning-driven Multiscale MD Workflows: The Mini-MuMMI Experience
Computational models have become one of the prevalent methods to model complex phenomena. To accurately model complex interactions, such as detailed biomolecular interactions, scientists often rely on multiscale models c…
Performance and Power Modeling and Prediction Using MuMMI and Ten Machine Learning Methods
In this paper, we use modeling and prediction tool MuMMI (Multiple Metrics Modeling Infrastructure) and ten machine learning methods to model and predict performance and power and compare their prediction error rates. We…
BIG-bench Machine LearningCPUPredictionEnhanced Multimodal Representation Learning with Cross-modal KD
This paper explores the tasks of leveraging auxiliary modalities which are only available at training to enhance multimodal representation learning through cross-modal Knowledge Distillation (KD). The widely adopted mutu…
Contrastive LearningEmotion ClassificationKnowledge DistillationRepresentation Learning+3Balanced Multimodal Learning via Mutual Information
Multimodal learning has increasingly become a focal point in research, primarily due to its ability to integrate complementary information from diverse modalities. Nevertheless, modality imbalance, stemming from factors …
Knowledge DistillationMutual Information Regularization for Weakly-supervised RGB-D Salient Object Detection
In this paper, we present a weakly-supervised RGB-D salient object detection model via scribble supervision. Specifically, as a multimodal learning task, we focus on effective multimodal representation learning via inter…
Objectobject-detectionObject DetectionPrediction+3