CARD: Diagnosing Belief to Action Routing Failures in Vision Language Models

2026-08-21

AI Summary

本文通过引入CARD方法，诊断视觉语言模型在将信念表示用于行动预测时的路由失败问题。

Abstract

Linear probes and activation steering have uncovered that vision-language models (VLMs) internally represent mental states such as agents' beliefs, knowledge, and intentions. However, it is unclear whether and how these representations are used by downstream predictions along these axes. To close this gap, we introduce Cross-Axis Routing Diagnostic (CARD), which steers activations along one axis while measuring the response of a different axis's prediction. Applied to open-weight VLMs on Relay Chain -- a new cooperative grid-world benchmark we propose -- we diagnose a critical routing failure: models fail to incorporate belief representations into their next action prediction, effectively leaving valuable information about their partners unused.

Problem

• vision-language models
• belief representations
• downstream predictions
• routing failure

Innovation

• Cross-Axis Routing Diagnostic
• belief representations
• vision-language models
• activation steering