Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

2026-07-03

AI Summary

Existing vision–language alignment methods exhibit fragility and overconfidence when confronted with synonymous paraphrases or text descriptions lacking fine-grained details. This work proposes a core–residual alignment framework that, for the first time, treats multi-view textual inputs as partially supervised signals. It distills a consensus semantic core as the alignment target while explicitly suppressing reliance on unexpressed residual information. Coupled with uncertainty-aware contrastive learning, the approach enhances representation robustness under weak supervision. Evaluated within a standard single-query architecture, the method achieves strong performance: 81.42% clean and 64.05% robust accuracy on ImageNet, 76.19% and 52.03% on the Avg-14 transfer benchmark, and 85.16% POPE F1 score and 59.57% OK-VQA accuracy when integrated with LLaVA-1.5-7B.

Abstract

Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly confident under paraphrase and omitted details. We aim to learn representations whose matching is stable across caption views and whose confidence reflects how strongly text constrains an image. We propose Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision. It distills a consensus semantic core as the alignment target, learns a single-view core predictor for standard inference with one query, and explicitly discourages vision-language similarity from depending on the orthogonal unsaid residual. An uncertainty-aware contrastive objective further softens alignment when caption views disagree, reducing overconfident updates under weak language constraints. Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite, while improving LVLM transfer with 85.16 POPE F1 and 59.57 OKVQA accuracy under an LLaVA-1.5-7B stack. These results suggest that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations under underspecified language supervision.

Problem

• vision-language alignment
• underspecified captions
• robustness
• partial constraint
• overconfidence

Innovation

• core-residual alignment
• partial constraint
• vision-language robustness
• uncertainty-aware contrastive learning
• underspecified supervision