Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources

2026-07-03

AI Summary

This work addresses the susceptibility of vision-language models to media brand cues—such as mastheads and logos—that induce content-agnostic priors in news credibility assessment. The authors introduce the CueTrust benchmark and the Source-Override Index (SOI), leveraging sparse autoencoders, inter-layer interventions, and cross-model diagnostics to mechanistically reveal, for the first time, that this prior exhibits a dual-encoding structure, is spatially localizable, and consistently replicable across models. Experiments demonstrate that brand cues can significantly modulate credibility judgments across an 11 log-odds range (ρ = 0.88). Targeted interventions reduce the source-overriding effect by 41% and generalize effectively to unseen media outlets, highlighting both the vulnerability and mitigability of such spurious associations in multimodal trust evaluation.

Abstract

Vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present. We show that VLMs carry a strong source-credibility prior keyed on outlet identity, and study it along three axes. (i) Cross-model benchmark. We introduce CueTrust, a cross-model diagnostic that measures which surface source cue overrides an article's content evidence via a Source-Override Index (SOI). Across seven VLMs and five cues, the vulnerability profile is model- and scale-dependent, and the override is outlet-identity-specific and encoding-invariant, firing from the masthead name, the logo image, or the bare domain, but not from a named author, in-text authority, or page layout (clean negative controls). (ii) Mechanistic account. For the brand cue, we give a full mechanistic account: swapping only the masthead moves credibility across an approximately 11 log-odds range that tracks professional ratings (rho = 0.88 with Media Bias/Fact Check). The prior is dual-coded (name and logo), strengthens with scale, is causally formed at layers 19-21, carried by interpretable seed-stable sparse-autoencoder features, and recurs at the same relative locus in a second model family. It overrides content (about 1.8x) as a signal-magnitude effect within a shared pathway, not a privileged route. Steering the localized direction selectively reduces the override (41% reduction) and generalizes to held-out outlets, confirming the prior is causally used, not merely decodable. Deployed VLMs may thus defer to source identity over the evidence in front of them, a reliability failure we can measure across models, localize, and causally probe. We release the stimulus suite and CueTrust.

Problem

• vision-language models
• source credibility
• news trustworthiness
• media bias
• credibility prior

Innovation

• vision-language models
• source credibility prior
• mechanistic interpretability
• CueTrust
• sparse autoencoders