Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

2026-04-10

AI Summary

This work addresses the vulnerability of multimodal large language models to malicious queries that elicit unsafe content, a challenge inadequately mitigated by existing defenses due to their limited precision in controlling specific harmful concepts. To overcome this, the authors propose a novel activation intervention framework that leverages a large-scale multimodal dictionary encompassing 15,000 concepts and sparse autoencoders (SAEs) to perform semantically interpretable, atomic-level interventions on frozen model activations during inference. This approach enables, for the first time, fine-grained, flexible, and interpretable suppression of harmful content. Extensive evaluations on models such as Qwen-VL, LLaVA, and InternVL, and benchmarks including MM-SafetyBench and JailBreakV, demonstrate substantial improvements in safety without compromising general capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response classification, or finetuning to improve MLLM safety. Nevertheless, such approaches are often ineffective against evolving malicious patterns, may require rerunning the query, or demand heavy computational resources. Steering the activations of a frozen model at inference time has recently emerged as a flexible and effective solution. However, existing steering methods for MLLMs typically handle only a narrow set of safety-related concepts or struggle to adjust specific concepts without affecting others. To address these challenges, we introduce Dictionary-Aligned Concept Control (DACO), a framework that utilizes a curated concept dictionary and a Sparse Autoencoder (SAE) to provide granular control over MLLM activations. First, we curate a dictionary of 15,000 multimodal concepts by retrieving over 400,000 caption-image stimuli and summarizing their activations into concept directions. We name the dataset DACO-400K. Second, we show that the curated dictionary can be used to intervene activations via sparse coding. Third, we propose a new steering approach that uses our dictionary to initialize the training of an SAE and automatically annotate the semantics of the SAE atoms for safeguarding MLLMs. Experiments on multiple MLLMs (e.g., QwenVL, LLaVA, InternVL) across safety benchmarks (e.g., MM-SafetyBench, JailBreakV) show that DACO significantly improves MLLM safety while maintaining general-purpose capabilities.

Problem

• Multimodal Large Language Models
• safety
• concept control
• malicious queries
• activation steering

Innovation

• Dictionary-Aligned Concept Control
• Sparse Autoencoder
• Activation Steering
• Multimodal Safety
• Concept Dictionary