Multi-Objective Optimization for Synthetic-to-Real Style Transfer

2026-02-03

AI Summary

This work addresses the domain gap between synthetic and real images by formulating style transfer as a sequence optimization problem, explicitly cast for the first time as a sequential decision-making task amenable to evolutionary search. The authors propose an efficient evaluation strategy based on single paired images and integrate it with a multi-objective genetic algorithm to simultaneously preserve structural consistency and enhance stylistic similarity to the target domain, substantially reducing search cost. The resulting augmentation pipelines demonstrate remarkable adaptability and diversity on cross-domain semantic segmentation tasks, notably GTA5→Cityscapes and ACDC, leading to significant performance improvements of downstream models in complex real-world scenarios.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Evolutionary Computation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Semantic segmentation networks require large amounts of pixel-level annotated data, which are costly to obtain for real-world images. Computer graphics engines can generate synthetic images alongside their ground-truth annotations. However, models trained on such images can perform poorly on real images due to the domain gap between real and synthetic images. Style transfer methods can reduce this difference by applying a realistic style to synthetic images. Choosing effective data transformations and their sequence is difficult due to the large combinatorial search space of style transfer operators. Using multi-objective genetic algorithms, we optimize pipelines to balance structural coherence and style similarity to target domains. We study the use of paired-image metrics on individual image samples during evolution to enable rapid pipeline evaluation, as opposed to standard distributional metrics that require the generation of many images. After optimization, we evaluate the resulting Pareto front using distributional metrics and segmentation performance. We apply this approach to standard datasets in synthetic-to-real domain adaptation: from the video game GTA5 to real image datasets Cityscapes and ACDC, focusing on adverse conditions. Results demonstrate that evolutionary algorithms can propose diverse augmentation pipelines adapted to different objectives. The contribution of this work is the formulation of style transfer as a sequencing problem suitable for evolutionary optimization and the study of efficient metrics that enable feasible search in this space. The source code is available at: https://github.com/echigot/MOOSS.

Problem

• domain gap
• style transfer
• synthetic-to-real
• semantic segmentation
• data transformation

Innovation

• multi-objective optimization
• style transfer
• evolutionary algorithms
• domain adaptation
• synthetic-to-real