VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models

2026-06-25

AI Summary

Traditional optical design prioritizes image quality, rendering it ill-suited for size-constrained scenarios and misaligned with the recognition objectives of frozen vision-language models. This work proposes CODA, a co-design framework that, for the first time, directly aligns the task objective of a meta-optical front-end with frozen zero-shot classifiers—such as CLIP, SigLIP, and DINOv2—eschewing conventional image reconstruction and fidelity constraints. The framework enables end-to-end training of continuous-density meta-optics through differentiable imaging and adjoint-gradient optimization grounded in Maxwell’s equations. Evaluated on ImageNet-100, the approach boosts zero-shot top-1 accuracy of CLIP ViT-L/14 from 53.75% to 65.41%. Moreover, the optimized meta-optical devices exhibit strong cross-model and cross-dataset transferability without requiring retraining.

Abstract

Conventional machine-vision pipelines typically rely on high-quality optics that produce clean, human-interpretable images, and optical design has therefore been driven by image-level criteria such as resolution, aberration correction, and pixel fidelity. However, such optics are often impractical for size-, cost-, or form-factor-constrained applications, where compact meta-optics offer an attractive alternative but operate under strict physical efficiency limits. We propose CODA, a co-design framework that optimizes a continuous-density meta-optic front-end for frozen-model recognition using differentiable image formation and adjoint-gradient updates of Maxwell-based simulations. CODA directly optimizes the cross-entropy loss of a fixed zero-shot CLIP classifier without learned reconstruction, image signal processing, or image-fidelity auxiliary objectives. In a two-dimensional simulated imaging benchmark on ImageNet-100, CODA improves CLIP ViT-L/14 zero-shot accuracy from 53.75 $\pm$ 3.57$\%$ with a focal-concentration baseline to 65.41 $\pm$ 3.99$\%$. The optimized optics further transfer without re-optimization across CLIP, SigLIP, and DINOv2 on ImageNet-100, CIFAR-100, and Food-101. These results demonstrate that, under constrained meta-optic imaging, downstream recognition can be improved by aligning optical design with frozen vision-model objectives rather than conventional image-formation criteria.

Problem

• meta-optics
• frozen vision-language models
• optical design
• image formation
• downstream recognition

Innovation

• meta-optics
• co-design
• frozen vision-language models
• differentiable imaging
• adjoint optimization