Harnessing Textual Refusal Directions for Multimodal Safety

2026-06-30

AI Summary

Existing multimodal large language models lack safety alignment methods that do not require multimodal harmful data. This work proposes MARS, the first training-free approach for cross-modal safety steering. MARS leverages a text-only rejection direction and performs re-centering in the activation space, adaptive intensity scaling, and optimal layer selection to effectively mitigate modality misalignment. Coupled with geometric trust-region constraints, it ensures robust cross-modal alignment. Experiments across five state-of-the-art multimodal large language models demonstrate that MARS significantly enhances safety under both image and video inputs, preserves task performance, and effectively defends against video-based jailbreaking attacks.

Abstract

To improve safety in Large Language Models (LLMs) we can either perform post-training alignment or exploit refusal directions in the activation space. Both strategies are less feasible in Multimodal LLMs (MLLMs) as they require unsafe multimodal data, harder to collect than their unimodal counterpart. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a light-weight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation re-centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak benchmarks, MARS achieves consistent safety gains while preserving utility. These results reveal that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.

Problem

• Multimodal Safety
• Refusal Directions
• Large Language Models
• Modality Alignment
• Safety Alignment

Innovation

• refusal directions
• multimodal safety
• training-free alignment
• activation steering
• modality-agnostic