Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting

2026-03-29

AI Summary

Existing methods for automatic oil painting generation often produce repetitive and unremarkable brushstrokes that lack artistic expressiveness. This work proposes a Differential Query Transformer architecture that emulates the human “observe–compare–paint” process by analyzing differential images to guide the model’s attention toward incremental changes between successive brushstrokes. By integrating differential image representations, positional encoding, and adversarial training, the approach significantly enhances sensitivity to local details and improves brushstroke prediction accuracy. Experimental results demonstrate that the proposed method generates visually realistic and artistically compelling oil paintings using fewer brushstrokes, consistently outperforming state-of-the-art techniques in both user studies and qualitative evaluations.

Abstract

This work introduces a new approach to automatic oil painting that emphasizes the creation of dynamic and expressive brushstrokes. A pivotal challenge lies in mitigating the duplicate and common-place strokes, which often lead to less aesthetic outcomes. Inspired by the human painting process, \ie, observing, comparing, and drawing, we incorporate differential image analysis into a neural oil painting model, allowing the model to effectively concentrate on the incremental impact of successive brushstrokes. To operationalize this concept, we propose the Differential Query Transformer (DQ-Transformer), a new architecture that leverages differentially derived image representations enriched with positional encoding to guide the stroke prediction process. This integration enables the model to maintain heightened sensitivity to local details, resulting in more refined and nuanced stroke generation. Furthermore, we incorporate adversarial training into our framework, enhancing the accuracy of stroke prediction and thereby improving the overall realism and fidelity of the synthesized paintings. Extensive qualitative evaluations, complemented by a controlled user study, validate that our DQ-Transformer surpasses existing methods in both visual realism and artistic authenticity, typically achieving these results with fewer strokes. The stroke-by-stroke painting animations are available on our project website.

Problem

• automatic oil painting
• brushstroke redundancy
• aesthetic quality
• expressive strokes
• painting realism

Innovation

• Differential Query Transformer
• automatic oil painting
• differential image analysis
• adversarial training
• stroke prediction