Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades

2026-06-13

AI Summary

This work addresses a critical vulnerability in cascaded multimodal large language model systems, where a weak model’s confidence score determines whether to route inputs to a more capable but costly strong model. The authors propose the Forced Delay Attack (FDA), a novel adversarial paradigm that manipulates routing decisions without altering answer correctness. FDA employs a universal border trigger added to input images and optimizes a temperature-smoothed objective to flatten the weak model’s output distribution, thereby systematically reducing its confidence and forcing frequent invocation of the strong model. This approach uniquely exploits the fragility of resource allocation mechanisms in cascaded architectures. Extensive experiments demonstrate that FDA consistently increases strong-model usage across diverse datasets, model architectures, and latency metrics, outperforming baseline methods such as image perturbations and prompt injection attacks.

Abstract

While multimodal large language models (MLLMs) have shown strong visual reasoning abilities, serving a large model for every query is computationally expensive. MLLM cascades mitigate this cost by first querying a weak but cheaper model and deferring to a strong model when the weak model's output is unconfident. However, since the weak model's confidence directly controls compute allocation, these systems expose a new attack surface: an adversary can manipulate confidence so that their queries are consistently deferred to the strong model. Motivated by this vulnerability, we introduce the Forced Deferral Attack (FDA), an adversarial image attack that lowers the weak model's confidence and causes cascades to route queries to the strong model. FDA learns a universal border trigger by optimizing a temperature-flattened objective. This objective pushes the weak model's token distribution on triggered inputs toward less concentrated targets constructed from its clean responses. Across datasets, model families, and deferral metrics, FDA consistently increases strong-model routing while outperforming image-perturbation and prompt-injection baselines. These results show that MLLM cascades are vulnerable to attacks that manipulate compute allocation, forcing unintended strong-model usage without directly targeting answer correctness.

Problem

• Forced Deferral Attack
• Multimodal LLM Cascades
• Confidence Manipulation
• Adversarial Image Attack
• Compute Allocation

Innovation

• Forced Deferral Attack
• multimodal LLM cascades
• adversarial image attack
• confidence manipulation
• compute allocation