How a Bit Becomes a Story: Semantic Steering via Differentiable Fault Injection

2025-12-08

AI Summary

This work investigates how hardware-level single-bit flips can be leveraged to *semantically steer* vision-language model (VLM) image captioning outputs—while preserving grammatical correctness. We propose the first *semantic-differentiable fault injection* paradigm, treating bit perturbations as trainable, semantics-aware operators—moving beyond conventional fault analysis focused solely on accuracy degradation. Based on gradient sensitivity estimation, we introduce BLADE, a differentiable fault analysis framework that jointly optimizes semantic consistency and linguistic fluency of generated captions, enabling bit-level differentiation over VLM weights. Experiments demonstrate that injecting a single-bit flip suffices for fine-grained semantic steering (e.g., “dog” → “wolf”) and systematic narrative alteration. Our findings uncover the encoding structure and plasticity of semantic information at the weight-bit level in large language models, establishing a novel paradigm for robustness evaluation, adversarial defense, and interpretable AI.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Large-scale security measurements

Abstract

Hard-to-detect hardware bit flips, from either malicious circuitry or bugs, have already been shown to make transformers vulnerable in non-generative tasks. This work, for the first time, investigates how low-level, bitwise perturbations (fault injection) to the weights of a large language model (LLM) used for image captioning can influence the semantic meaning of its generated descriptions while preserving grammatical structure. While prior fault analysis methods have shown that flipping a few bits can crash classifiers or degrade accuracy, these approaches overlook the semantic and linguistic dimensions of generative systems. In image captioning models, a single flipped bit might subtly alter how visual features map to words, shifting the entire narrative an AI tells about the world. We hypothesize that such semantic drifts are not random but differentiably estimable. That is, the model's own gradients can predict which bits, if perturbed, will most strongly influence meaning while leaving syntax and fluency intact. We design a differentiable fault analysis framework, BLADE (Bit-level Fault Analysis via Differentiable Estimation), that uses gradient-based sensitivity estimation to locate semantically critical bits and then refines their selection through a caption-level semantic-fluency objective. Our goal is not merely to corrupt captions, but to understand how meaning itself is encoded, distributed, and alterable at the bit level, revealing that even imperceptible low-level changes can steer the high-level semantics of generative vision-language models. It also opens pathways for robustness testing, adversarial defense, and explainable AI, by exposing how structured bit-level faults can reshape a model's semantic output.

Problem

• Investigates bit-level fault injection effects on LLM semantics
• Explores how bit flips alter semantic meaning in image captioning
• Develops gradient-based method to locate semantically critical bits

Innovation

• Differentiable fault injection analyzes bit-level semantic sensitivity
• Gradient-based estimation locates bits influencing meaning while preserving grammar
• BLADE framework identifies critical bits through semantic-fluency objectives