Positional Encoding Field

2025-10-23

AI Summary

Existing Diffusion Transformers (DiTs) rely on 2D positional encodings (PEs), limiting their ability to model 3D geometry and hindering performance in depth-aware tasks such as single-image novel view synthesis and controllable spatial editing. This work identifies the dominant role of PEs in DiT-based visual generation and introduces Positional Encoding Fields (PE-Fields): a structured, 3D PE representation that explicitly encodes depth and hierarchical spatial sub-block relationships, enabling DiTs to directly model volumetric geometry in 3D space. Our method integrates perturbation sensitivity analysis, principled 3D PE design, and a hierarchical encoding mechanism. Experiments demonstrate state-of-the-art performance on single-image novel view synthesis and, for the first time, enable fine-grained, geometrically consistent spatial editing. PE-Fields establish a scalable, geometry-aware encoding paradigm for enhancing the 3D perception capabilities of DiTs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Diffusion Transformers (DiTs) have emerged as the dominant architecture for visual generation, powering state-of-the-art image and video models. By representing images as patch tokens with positional encodings (PEs), DiTs combine Transformer scalability with spatial and temporal inductive biases. In this work, we revisit how DiTs organize visual content and discover that patch tokens exhibit a surprising degree of independence: even when PEs are perturbed, DiTs still produce globally coherent outputs, indicating that spatial coherence is primarily governed by PEs. Motivated by this finding, we introduce the Positional Encoding Field (PE-Field), which extends positional encodings from the 2D plane to a structured 3D field. PE-Field incorporates depth-aware encodings for volumetric reasoning and hierarchical encodings for fine-grained sub-patch control, enabling DiTs to model geometry directly in 3D space. Our PE-Field-augmented DiT achieves state-of-the-art performance on single-image novel view synthesis and generalizes to controllable spatial image editing.

Problem

• Investigating patch token independence in Diffusion Transformers
• Extending positional encodings to structured 3D fields
• Enabling 3D geometry modeling for view synthesis and editing

Innovation

• Extends positional encodings to structured 3D field
• Incorporates depth-aware encodings for volumetric reasoning
• Uses hierarchical encodings for fine-grained sub-patch control