Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting

2026-03-01

AI Summary

This work addresses the challenge of generating expressive and controllable oil-painting strokes under extremely limited data conditions—specifically, with only 470 hand-drawn samples—where existing methods struggle to achieve satisfactory performance. To this end, we propose StrokeDiff, a novel framework that enhances diffusion model stability under sparse supervision by injecting random visual priors via Smooth Regularization (SmR) during training. Additionally, a Bézier curve-based conditioning module is introduced to enable structured and controllable stroke generation. The framework seamlessly integrates prediction, generation, ranking, and composition into an end-to-end painting pipeline. Experimental results demonstrate that our approach produces strokes with both high diversity and structural coherence, significantly improving textural richness and depth. Its superiority is consistently validated through both quantitative metrics and human evaluations.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Many creative multimedia systems are built upon visual primitives such as strokes or textures, which are difficult to collect at scale and fundamentally different from natural image data. This data scarcity makes it challenging for modern generative models to learn expressive and controllable primitives, limiting their use in process-aware content creation. In this work, we study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples (n=470) and propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR). SmR injects stochastic visual priors during training, providing a simple mechanism to stabilize diffusion models under sparse supervision without altering the inference process. We further show how the learned primitives can be made controllable through a Bézier-based conditioning module and integrated into a complete stroke-based painting pipeline, including prediction, generation, ordering, and compositing. This demonstrates how data-efficient primitive modeling can support expressive and structured multimedia content creation. Experiments indicate that the proposed approach produces diverse and structurally coherent brushstrokes and enables paintings with richer texture and layering, validated by both automatic metrics and human evaluation.

Problem

• brushstroke generation
• data efficiency
• diffusion models
• oil painting
• visual primitives

Innovation

• diffusion models
• data-efficient learning
• brushstroke generation
• smooth regularization
• Bézier-based conditioning