State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping
Choosing an appropriate representation of the environment for the underlying decision-making process of the reinforcement learning agent is not always straightforward. The state representation should be inclusive enough to allow the agent to informatively decide on its actions and disentangled enough to simplify policy training and the corresponding sim2real transfer. Given this outlook, this work examines the effect of various representations in incentivizing the agent to solve a specific robotic task: antipodal and planar object grasping. A continuum of state representations is defined, starting from hand-crafted numerical states to encoded image-based representations, with decreasing levels of induced task-specific knowledge. The effects of each representation on the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot are examined and compared against a model-based approach with complete system knowledge. The results show that reinforcement learning agents using numerical states can perform on par with non-learning baselines. Furthermore, we find that agents using image-based representations from pre-trained environment embedding vectors perform better than end-to-end trained agents, and hypothesize that separation of representation learning from reinforcement learning can benefit sim2real transfer. Finally, we conclude that incentivizing the state representation with task-specific knowledge facilitates faster convergence for agent training and increases success rates in sim2real robot control.
Code (1)
Tasks
Decision Makingreinforcement-learningReinforcement LearningRepresentation LearningRobotic GraspingSimilar Papers 제목 키워드 기반
Penalizing side effects using stepwise relative reachability
How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any i…
Reinforcement LearningSafe Reinforcement LearningExploration and Incentives in Reinforcement Learning
How do you incentivize self-interested agents to $\textit{explore}$ when they prefer to $\textit{exploit}$? We consider complex exploration problems, where each agent faces the same (but unknown) MDP. In contrast with tr…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Learning to Learn Group Alignment: A Self-Tuning Credo Framework with Multiagent Teams
Mixed incentives among a population with multiagent teams has been shown to have advantages over a fully cooperative system; however, discovering the best mixture of incentives or team structure is a difficult and dynami…
Hierarchical Reinforcement LearningMeta-LearningLearning Homophilic Incentives in Sequential Social Dilemmas
Promoting cooperation among self-interested agents is a long-standing and interdisciplinary problem, but receives less attention in multi-agent reinforcement learning (MARL). Game-theoretical studies reveal that altruist…
Multi-agent Reinforcement LearningSegregation Dynamics with Reinforcement Learning and Agent Based Modeling
Societies are complex. Properties of social systems can be explained by the interplay and weaving of individual actions. Incentives are key to understand people's choices and decisions. For instance, individual preferenc…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)