🤖 AI Summary
This work addresses the challenge that existing vision-language models struggle to adapt in in-context learning scenarios where task semantics remain constant but decision criteria dynamically shift. To tackle this, the paper introduces a novel paradigm termed Criterion-Conditioned In-Context Learning (CC-ICL), which requires models to infer implicit decision criteria from provided examples and adjust predictions accordingly. The authors construct the first CC-ICL benchmark, CC-Bench, featuring a dual-layer data structure spanning multiple domains, along with newly proposed metrics—criterion invariance and sensitivity—to evaluate model robustness and adaptability under criterion shifts. Experiments reveal that prevailing models exhibit rigid decision boundary biases, yet simple multi-criterion training substantially enhances criterion sensitivity in 7B-scale models, outperforming closed-source counterparts without compromising general multimodal capabilities.
📝 Abstract
Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction. In the standard ICL setting, once the task is induced, its decision criterion remains fixed. However, in real-world applications, many tasks exhibit a stable high-level intent, while their decision criteria shift according to specific requirements. Thus, we introduce a new setting, denoted as Criterion-Conditional In-Context Learning (CC-ICL), where models must infer the latent criterion from context and adjust predictions accordingly under fixed task semantics. To evaluate this capability, we propose two complementary metrics, Criterion Invariance and Criterion Sensitivity, capturing the model's robustness and adaptability under criterion shifts. We further construct CC-Bench, a multi-domain benchmark that supports evaluation under the CC-ICL setting. By employing a dual-level data hierarchy, CC-Bench enables legitimate ground-truth variation conditioned on the active criterion even when the task remains fixed. Experiments on CC-Bench reveal that most models exhibit a rigid boundary bias, struggling to align their decisions with the latent criterion. We also find that even a simple multi-criterion training strategy can significantly reduce this bias, improving Criterion Sensitivity and enabling 7B-scale models to surpass proprietary models without degrading general multimodal performance.