Communicability-Inspired Positional Encoding (CIPE)

2026-06-23

AI Summary

Designing position encodings that are geometrically compatible with attention mechanisms on non-Euclidean graphs remains challenging. This work proposes a novel approach that constructs position encodings based on pairwise node communicability, translating full-path connectivity into a similarity geometry suitable for self-attention. By employing a dimensionality-alignment mapping, the method adapts to arbitrary model dimensions while ensuring that inner products directly reflect structural relatedness, thereby achieving consistency between attention and graph geometry. Evaluated across seven benchmark datasets, the proposed encoding improves the performance of structure-agnostic Transformers by an average of 35.5% and consistently enhances the effectiveness of structure-aware graph Transformers.

Abstract

Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging. Such encodings should ideally induce an Attention-Compatible Geometry for self-attention: not merely describing graph structure, but defining a geometry whose inner products reflect meaningful structural relatedness. To realize this geometry, we propose Communicability-Inspired Positional Encoding (CIPE), built from communicability, a measure between pairs of nodes that aggregates contributions from paths of all lengths. By construction, CIPE inner products recover communicability, converting global multi-path connectivity into an attention-ready similarity geometry. For practical Transformer training, we introduce dimensionality alignment, mapping graph-size-dependent CIPE representations to prescribed dimensions while faithfully preserving the induced geometry. Empirically, CIPE improves structure-agnostic Transformers by 35.5% on average across seven benchmarks, outperforming representative PEs; it also consistently improves structure-biased graph Transformers, where competing PEs often yield only marginal benefits. These results position CIPE as a principled framework for attention-compatible graph positional encodings.

Problem

• positional encoding
• non-Euclidean graphs
• self-attention
• attention-compatible geometry
• graph structure

Innovation

• Communicability
• Positional Encoding
• Attention-Compatible Geometry
• Graph Transformers
• Dimensionality Alignment