Communicability-Inspired Positional Encoding (CIPE)
Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging. Such encodings should ideally induce an Attention-Compatible Geometry for self-attention: not merely describing graph structure, but defining a geometry whose inner products reflect meaningful structural relatedness. To realize this geometry, we propose Communicability-Inspired Positional Encoding (CIPE), built from communicability, a measure between pairs of nodes that aggregates contributions from paths of all lengths. By construction, CIPE inner products recover communicability, converting global multi-path connectivity into an attention-ready similarity geometry. For practical Transformer training, we introduce dimensionality alignment, mapping graph-size-dependent CIPE representations to prescribed dimensions while faithfully preserving the induced geometry. Empirically, CIPE improves structure-agnostic Transformers by 35.5% on average across seven benchmarks, outperforming representative PEs; it also consistently improves structure-biased graph Transformers, where competing PEs often yield only marginal benefits. These results position CIPE as a principled framework for attention-compatible graph positional encodings.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers
The remarkable scalability of Transformers has expanded their application to 3D computer vision, where camera-aware positional encoding is crucial for providing spatial cues in multi-view geometry. Recent advancements ha…
Novel View SynthesisRecipe for a General, Powerful, Scalable Graph Transformer
We propose a recipe on how to build a general, powerful, scalable (GPS) graph Transformer with linear complexity and state-of-the-art results on a diverse set of benchmarks. Graph Transformers (GTs) have gained popularit…
Graph ClassificationGraph Property PredictionGraph RegressionGraph Representation Learning+4Efficient Training of Diffusion Mixture-of-Experts Models: A Practical Recipe
Recent efforts on Diffusion Mixture-of-Experts (MoE) models have primarily focused on developing more sophisticated routing mechanisms. However, we observe that the underlying architectural configuration space remains ma…
Signed graphs in data sciences via communicability geometry
Signed graphs are an emergent way of representing data in a variety of contexts where antagonistic interactions exist. These include data from biological, ecological, and social systems. Here we propose the concept of co…
Dimensionality ReductionAnalyzing Communicability and Connectivity in the Indian Stock Market During Crises
In financial networks, information does not always follow the shortest path between two nodes but may also take alternate routes. Communicability, a network measure, resolves this complexity and, in diffusion-like proces…