Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation

2026-06-22

AI Summary

This study addresses the challenge of enhancing perception robustness for autonomous driving under adverse weather conditions by jointly optimizing on-vehicle camera sensor design and downstream semantic segmentation. The authors propose a differentiable end-to-end training framework operating in the RAW domain to systematically investigate the impact of spectral color filter array (CFA) configurations, point spread function (PSF), and noise models on dense prediction performance. For the first time, they quantitatively assess how sensor degrees of freedom influence segmentation accuracy, revealing that learning 2×2 CFA weights yields significant gains, whereas increasing CFA size or optimizing the PSF is detrimental. The method achieves consistent improvements of 1.7% and 2.3% in mIoU on KITTI-360 and ACDC benchmarks, respectively, and demonstrates model-agnostic robustness across fog, rain, snow, and nighttime scenarios.

Abstract

Robust perception underpins autonomous driving, and most recent progress comes from scaling the model-larger backbones, foundation models, and cooperative multi-agent fusion. We pursue a complementary, upstream question: what should the camera itself measure? Using a differentiable RAW-to-task pipeline, we decompose which sensor degrees of freedom benefit dense prediction. Learning the spectral colour-filter-array (CFA) weights is the dominant lever, improving mIoU by +0.017 (KITTI-360) and +0.023 (ACDC) over a fixed camera. In contrast, point-spread-function (optics) co-design is net-negative (-0.020 mIoU on KITTI-360) - a consequence of the data-processing inequality, which also bounds the task information that any downstream model, however large or cooperative, can recover. Noise co-optimisation is marginal, and counter to intuition enlarging the CFA tile beyond 2x2 consistently hurts, as the filters are confined to the rank three sRGB input. Because the intervention is at the sensor, the gains are model-agnostic; we validate robustness on ACDC's fog, night, rain, and snow, and conclude with a simple recipe: learn the 2x2 CFA weights and keep an identity PSF.

Problem

• sensor co-design
• autonomous driving
• semantic segmentation
• color filter array
• robust perception

Innovation

• sensor co-design
• color filter array
• differentiable pipeline
• autonomous driving perception
• task-optimal imaging