Causal Influence Detection for Improving Efficiency in Reinforcement Learning
Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of \emph{situation-dependent causal influence} based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
Code (2)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
CAIMAN: Causal Action Influence Detection for Sample-efficient Loco-manipulation
Enabling legged robots to perform non-prehensile loco-manipulation is crucial for enhancing their versatility. Learning behaviors such as whole-body object pushing often requires sophisticated planning strategies or exte…
Causal-Aware Intelligent QoE Optimization for VR Interaction with Adaptive Keyframe Extraction
The optimization of quality of experience (QoE) in multi-user virtual reality (VR) interactions demands a delicate balance between ultra-low latency, high-fidelity motion synchronization, and equitable resource allocatio…
Causal InferenceCPUFairnessReinforcement Learning (RL)Variable-Agnostic Causal Exploration for Reinforcement Learning
Modern reinforcement learning (RL) struggles to capture real-world cause-and-effect dynamics, leading to inefficient exploration due to extensive trial-and-error actions. While recent efforts to improve agent exploration…
Causal Discoveryreinforcement-learningReinforcement LearningReinforcement Learning (RL)Causal Model-Based Reinforcement Learning for Sample-Efficient IoT Channel Access
Despite the advantages of multi-agent reinforcement learning (MARL) for wireless use case such as medium access control (MAC), their real-world deployment in Internet of Things (IoT) is hindered by their sample inefficie…
Multi-agent Reinforcement LearningData AugmentationSituation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning
Learning to collaborate has witnessed significant progress in multi-agent reinforcement learning (MARL). However, promoting coordination among agents and enhancing exploration capabilities remain challenges. In multi-age…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning