When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs

2026-07-08

AI Summary

This work addresses the issue of overconfidence in vision-language models during test-time adaptation via prompt tuning, which degrades calibration performance. To mitigate this, the paper introduces— for the first time—a symbolic attribute graph that represents (class, attribute) pairs as nodes and explicitly models structured dependencies among attributes by integrating graph attention networks with contrastive learning. Two attribute selection strategies are proposed: ARGTCA-DIV, which enhances intra-class diversity, and ARGTCA-DISC, which emphasizes inter-class discriminability. Evaluated across nine benchmarks, ARGTCA-DIV reduces the Expected Calibration Error (ECE) by approximately 37% on average, while ARGTCA-DISC achieves a reduction of about 17%, both significantly outperforming existing methods.

Abstract

Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.

Problem

• vision-language models
• calibration
• attribute reasoning
• graph structure
• test-time adaptation

Innovation

• Graph Attention Network
• Symbolic Attribute Graph
• Test-Time Adaptation
• Calibration
• Vision-Language Models