Mechanistically Interpreting Compression in Vision-Language Models

2026-03-26

AI Summary

Compressing vision-language models (VLMs) may compromise their internal computational mechanisms and safety behaviors, yet the underlying effects remain poorly understood. This work systematically investigates how pruning and quantization impact VLMs’ internal structures and their capacity for safe refusal, leveraging causal circuit analysis and cross-encoder feature alignment. We reveal for the first time that while pruning preserves circuit topology, it degrades genuine refusal capability; in contrast, quantization alters high-level circuitry but maintains feature alignment. Building on these insights, we introduce VLMSafe-420, the first benchmark specifically designed to evaluate safety risks introduced by VLM compression, effectively identifying security vulnerabilities arising from model compression techniques.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Compressed vision-language models (VLMs) are widely used to reduce memory and compute costs, making them a suitable choice for real-world deployment. However, compressing these models raises concerns about whether internal computations and safety behaviors are preserved. In this work, we use causal circuit analysis and crosscoder-based feature comparisons to examine how pruning and quantization fundamentally change the internals across representative VLMs. We observe that pruning generally keeps circuit structure intact but rotates and attenuates internal features, while quantization modifies the circuits at a higher level yet leaves the surviving features better aligned. Leveraging this insight, we also introduce VLMSafe-420, a novel benchmark that pairs harmful inputs with matched benign counterfactuals across various safety categories. Our findings show that pruning causes a sharp drop in genuine refusal behavior, suggesting that the choice of compression has safety implications.

Problem

• vision-language models
• model compression
• safety behavior
• pruning
• quantization

Innovation

• causal circuit analysis
• vision-language models
• model compression
• safety evaluation
• VLMSafe-420