Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation

2024-03-26

arXiv.org

AI Summary

Sketches exhibit unreliable temporal adjacency graph edges and inconsistent inter-block contextual relationships due to variations in drawing order. To address this, we propose a sketch representation learning method that does not rely on explicit drawing-order modeling. Our core innovation is a dual-path positional encoding mechanism: fixed sinusoidal absolute encoding captures global drawing order, while learnable relative encoding models semantic context among unseen blocks; both are injected into node representations rather than used for graph structure construction. Integrating GCN with a semantic proximity graph enables joint preservation of visual features and incorporation of global context. Extensive experiments on sketch completion and controllable synthesis demonstrate significant performance gains, validating the effectiveness and generalizability of our order-enhanced representation.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

When benefiting graphic sketch representation with sketch drawing orders, recent studies have linked sketch patches as graph edges by drawing orders in accordance to a temporal-based nearest neighboring strategy. However, such constructed graph edges may be unreliable, since the contextual relationships between patches may be inconsistent with the sequential positions in drawing orders, due to variants of sketch drawings. In this paper, we propose a variant-drawing-protected method by equipping sketch patches with context-aware positional encoding (PE) to make better use of drawing orders for sketch learning. We introduce a sinusoidal absolute PE to embed the sequential positions in drawing orders, and a learnable relative PE to encode the unseen contextual relationships between patches. Both types of PEs never attend the construction of graph edges, but are injected into graph nodes to cooperate with the visual patterns captured from patches. After linking nodes by semantic proximity, during message aggregation via graph convolutional networks, each node receives both semantic features from patches and contextual information from PEs from its neighbors, which equips local patch patterns with global contextual information, further obtaining drawing-order-enhanced sketch representations. Experimental results indicate that our method significantly improves sketch healing and controllable sketch synthesis. The source codes could be found at https://github.com/SCZang/DC-gra2seq.

Problem

• Unreliable graph edges from inconsistent sketch drawing orders
• Lack of contextual relationships in sequential patch positions
• Need for drawing-order-enhanced sketch representation learning

Innovation

• Context-aware positional encoding for sketch patches
• Sinusoidal and learnable positional encodings combined
• Graph nodes inject semantic and contextual features