Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images

2026-03-09

AI Summary

This work addresses the vulnerability of multimodal large language models to generating harmful content when processing visual inputs, a challenge exacerbated by the inadequacy of existing safety alignment methods that rely on explicit labels or contrastive data and struggle with abstract safety concepts. The paper proposes the first label-free, visually grounded self-supervised alignment approach: by fine-tuning models on neutral visual question answering tasks constructed from threat-related images, the method enables implicit internalization of caution and vigilance through repeated exposure, thereby shaping safety-oriented behaviors. This approach extends the self-supervision paradigm from text to vision for the first time, significantly reducing attack success rates, improving response quality, mitigating over-rejection, and preserving general capabilities across multiple vision-language models and safety benchmarks.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Multimodal large language models (MLLMs) face safety misalignment, where visual inputs enable harmful outputs. To address this, existing methods require explicit safety labels or contrastive data; yet, threat-related concepts are concrete and visually depictable, while safety concepts, like helpfulness, are abstract and lack visual referents. Inspired by the Self-Fulfilling mechanism underlying emergent misalignment, we propose Visual Self-Fulfilling Alignment (VSFA). VSFA fine-tunes vision-language models (VLMs) on neutral VQA tasks constructed around threat-related images, without any safety labels. Through repeated exposure to threat-related visual content, models internalize the implicit semantics of vigilance and caution, shaping safety-oriented personas. Experiments across multiple VLMs and safety benchmarks demonstrate that VSFA reduces the attack success rate, improves response quality, and mitigates over-refusal while preserving general capabilities. Our work extends the self-fulfilling mechanism from text to visual modalities, offering a label-free approach to VLMs alignment.

Problem

• safety alignment
• multimodal large language models
• visual inputs
• harmful outputs
• abstract safety concepts

Innovation

• Visual Self-Fulfilling Alignment
• safety alignment
• multimodal large language models
• threat-related images
• label-free fine-tuning