MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space

2026-05-19

AI Summary

While existing vision models achieve strong performance on standard benchmarks, the specific scene factors influencing their predictions remain poorly understood, and current robustness evaluations lack fine-grained control over 3D scene parameters. To address this, this work introduces the MAPS dataset, comprising the first large-scale, extensible 3D asset library validated with ImageNet categories, and develops a parametric rendering pipeline in Blender that enables continuous, independent manipulation of nine scene factors—including background, camera pose, and lighting. Through regression-based sensitivity analysis and large-scale synthetic image generation, the study systematically evaluates CNN and Transformer architectures, revealing that both exhibit high sensitivity to camera distance and elevation angle, and display convergent sensitivity patterns that markedly differ from earlier model designs.

Abstract

Modern vision models achieve strong performance on standard benchmarks, yet their aggregate accuracy reveals little about which scene properties drive their predictions. Existing robustness benchmarks provide important stress tests, but typically manipulate global 2D image properties, rely on entangled real-world variation, or cover only a limited set of 3D objects and scene parameters. We introduce MAPS (Manifolds of Artificial Parametric Scenes), a scalable instrument for controlled attribution of vision model behavior to scene parameters. MAPS comprises 2,618 curated photorealistic 3D meshes validated for recognizability across 560 ImageNet classes and provides a Blender-based rendering pipeline for on-demand image generation under continuous variation of nine independent scene-factors spanning background, camera, and lighting, extensible to other factors. To showcase its applicability, we use MAPS to evaluate 20 convolutional and transformer-based models by quantifying their reliance on these scene factors through regression-based sensitivity analysis. We find a near-universal failure axis across all tested architectures: camera distance and elevation consistently dominate recognition failure regardless of ImageNet accuracy. However, the full sensitivity structure reveals that modern CNNs and transformers cluster together, distinct from older architectures, suggesting that fine-grained architectural design choices, rather than the coarse CNN-versus-transformer distinction, are the stronger determinant of sensitivity profiles.

Problem

• vision models
• scene parameters
• 3D scenes
• model robustness
• controlled evaluation

Innovation

• controlled 3D scene space
• parametric scene generation
• scene-factor disentanglement
• vision model sensitivity analysis
• synthetic dataset for attribution