Space filling positionality and the Spiroformer
Transformers excel when dealing with sequential data. Generalizing transformer models to geometric domains, such as manifolds, we encounter the problem of not having a well-defined global order. We propose a solution with attention heads following a space-filling curve. As a first experimental example, we present the Spiroformer, a transformer that follows a polar spiral on the $2$-sphere.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Compositionality for Recursive Neural Networks
Modelling compositionality has been a longstanding area of research in the field of vector space semantics. The categorical approach to compositionality maps grammar onto vector spaces in a principled way, but comes unde…
Exploring Compositionality in Vision Transformers using Wavelet Representations
While insights into the workings of the transformer model have largely emerged by analysing their behaviour on language tasks, this work investigates the representations learnt by the Vision Transformer (ViT) encoder thr…
Representation LearningMeasuring the compositionality of noun-noun compounds over time
We present work in progress on the temporal progression of compositionality in noun-noun compounds. Previous work has proposed computational methods for determining the compositionality of compounds. These methods try to…
Compositionality and Generalization in Emergent Languages
Natural language allows us to refer to novel composite concepts by combining expressions denoting their parts according to systematic rules, a property known as \emph{compositionality}. In this paper, we study whether th…
DisentanglementRepresentation LearningVariants of Vector Space Reductions for Predicting the Compositionality of English Noun Compounds
Predicting the degree of compositionality of noun compounds such as {``}snowball{''} and {``}butterfly{''} is a crucial ingredient for lexicography and Natural Language Processing applications, to know whether the compou…