Relational inductive biases on attention mechanisms
Inductive learning aims to construct general models from specific examples, guided by biases that influence hypothesis selection and determine generalization capacity. In this work, we focus on characterizing the relational inductive biases present in attention mechanisms, understood as assumptions about the underlying relationships between data elements. From the perspective of geometric deep learning, we analyze the most common attention mechanisms in terms of their equivariance properties with respect to permutation subgroups, which allows us to propose a classification based on their relational biases. Under this perspective, we show that different attention layers are characterized by the underlying relationships they assume on the input data.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Grid-to-Graph: Flexible Spatial Relational Inductive Biases for Reinforcement Learning
Although reinforcement learning has been successfully applied in many domains in recent years, we still lack agents that can systematically generalize. While relational inductive biases that fit a task can improve genera…
reinforcement-learningReinforcement Learning (RL)R-SQAIR: Relational Sequential Attend, Infer, Repeat
Traditional sequential multi-object attention models rely on a recurrent mechanism to infer object relations. We propose a relational extension (R-SQAIR) of one such attention model (SQAIR) by endowing it with a module w…
Inductive BiasObjectGraph Machine: Exploring Edge Mechanisms as an Inductive Bias
Transformers provide a powerful architecture for global content-based matching, but reasoning problems may benefit from a stronger inductive bias toward iterative traversal of latent relations. We introduce Graph Machine…
Learning Hierarchical Relational Representations through Relational Convolutions
An evolving area of research in deep learning is the study of architectures and inductive biases that support the learning of relational feature representations. In this paper, we address the challenge of learning repres…
RelationDisentangling and Integrating Relational and Sensory Information in Transformer Architectures
Relational reasoning is a central component of generally intelligent systems, enabling robust and data-efficient inductive generalization. Recent empirical evidence shows that many existing neural architectures, includin…
Information RetrievalLanguage ModelingLanguage ModellingRelational Reasoning+1