DeepBias: Adaptive In-depth Probing of Social Biases in LVLMs

2026-07-13

AI Summary

Existing static datasets struggle to thoroughly evaluate social biases in large vision-language models (LVLMs). To address this limitation, this work proposes DeepBias, a novel framework that introduces a dynamic, adaptive paradigm for bias probing. DeepBias employs a ProposerAgent to generate initial test samples and a DiggerAgent that iteratively refines these samples through multi-round evolution, leveraging a multi-strategy skill repository and a response-conditioned rewriting mechanism. The agents’ behaviors are further optimized via Direct Preference Optimization (DPO), while ensemble anchoring across multiple models enhances generalization. The resulting benchmark, DeepBiasBench, effectively uncovers deep-seated biases in LVLMs and substantially outperforms static evaluation approaches, advancing bias assessment toward an evolutionary, interactive paradigm.

Abstract

While Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities, they remain highly susceptible to embedded social biases. Existing bias evaluation protocols predominantly rely on static datasets, which provide only a superficial assessment, as their fixed test cases cannot adaptively evolve to measure the true depth and limits of model vulnerabilities. We introduce DeepBias, an adaptive framework for the in-depth probing of social biases in LVLMs with carefully designed agents. Our approach operates through a dynamic ''generation-evolution-probing'' loop. First, a generative ProposerAgent synthesizes test data and is iteratively updated via Direct Preference Optimization (DPO) based on the target LVLM's responses, exploring model-specific failure modes. Second, an autonomous skill-driven DiggerAgent rewrites each test data across multiple probing turns, adaptively selecting from a curated skill library of deepening and rewriting strategies. At each turn, this process is conditioned on the model's previous response, enabling progressively deeper biases to be exposed. Furthermore, we build a benchmark named DeepBiasBench using our framework. By employing an ensemble of five diverse state-of-the-art LVLMs as anchors, the benchmark captures vulnerabilities shared across architectures. Comprehensive experiments demonstrate the effectiveness of our framework and show that DeepBias provides a challenging benchmark for in-depth bias evaluation, establishing an evolutionary paradigm for LVLM safety assessment.

Problem

• social bias
• Large Vision-Language Models
• bias evaluation
• adaptive probing
• model vulnerability

Innovation

• adaptive bias probing
• vision-language models
• Direct Preference Optimization
• dynamic evaluation framework
• social bias detection