Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?

2026-08-19

AI Summary

研究探讨了在部署偏移情况下，零样本视觉-语言模型的边际覆盖是否能保证类条件安全性，并通过多种方法测试其效果，但发现边际覆盖不能作为类尾部的安全保障。

Abstract

Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to $\approx 0$ and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3$\times$ cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level $α$, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.

Problem

• marginal coverage
• class-conditional safety
• zero-shot VLMs
• deployment shift
• tail coverage

Innovation

• marginal coverage
• class-conditional safety
• zero-shot VLMs
• deployment shift
• target-side class calibration