Minimal Neural Network Models for Permutation Invariant Agents

2022-05-12 · Joachim Winther Pedersen, Sebastian Risi

Organisms in nature have evolved to exhibit flexibility in face of changes to the environment and/or to themselves. Artificial neural networks (ANNs) have proven useful for controlling of artificial agents acting in environments. However, most ANN models used for reinforcement learning-type tasks have a rigid structure that does not allow for varying input sizes. Further, they fail catastrophically if inputs are presented in an ordering unseen during optimization. We find that these two ANN inflexibilities can be mitigated and their solutions are simple and highly related. For permutation invariance, no optimized parameters can be tied to a specific index of the input elements. For size invariance, inputs must be projected onto a common space that does not grow with the number of projections. Based on these restrictions, we construct a conceptually simple model that exhibit flexibility most ANNs lack. We demonstrate the model's properties on multiple control problems, and show that it can cope with even very rapid permutations of input indices, as well as changes in input size. Ablation studies show that is possible to achieve these properties with simple feedforward structures, but that it is much easier to optimize recurrent structures.

📄 PDF Abstract BibTeX arXiv:2205.07868

Code (0)

등록된 구현이 없습니다.

HARP: Human-Assisted Regrouping with Permutation Invariant Critic for Multi-Agent Reinforcement Learning

2024-09-18 · Huawen Hu, Enze Shi, Chenxi Yue, Shuocun Yang 외

Human-in-the-loop reinforcement learning integrates human expertise to accelerate agent learning and provide critical guidance and feedback in complex fields. However, many existing approaches focus on single-agent tasks…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

A Principled Permutation Invariant Approach to Mean-Field Multi-Agent Reinforcement Learning

2021-09-29 · Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang 외

Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents. To address such a chall…

Inductive Bias Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach

2021-05-18 · Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang 외

Inductive Bias Multi-agent Reinforcement Learning

PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learning

2019-10-31 · Iou-Jen Liu, Raymond A. Yeh, Alexander G. Schwing

Sample efficiency and scalability to a large number of agents are two important goals for multi-agent reinforcement learning systems. Recent works got us closer to those goals, addressing non-stationarity of the environm…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Minimal model of permutation symmetry in unsupervised learning

2019-04-30 · Tianqi Hou, K. Y. Michael Wong, Haiping Huang

Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class …

model