Selective Test-Time Debiasing for CLIP via Reward Gating

2026-07-01

AI Summary

This work addresses the tendency of vision-language models to propagate social stereotypes in person-related queries, a challenge exacerbated by existing debiasing methods that apply uniform corrections across all inputs, often compromising the trade-off between fairness and utility. To overcome this limitation, the authors propose RG-TTA, a reinforcement learning–based test-time adaptation framework that dynamically decides whether to apply fairness regularization based on the input’s sensitivity to bias. Integrating a reward-gated mechanism, test-time policy adaptation, and cross-modal alignment optimization, RG-TTA significantly reduces stereotypical biases on benchmarks such as FairFace and UTKFace while simultaneously enhancing zero-shot task performance, thereby effectively breaking the longstanding fairness–utility trade-off.

Abstract

Vision language models (VLMs) demonstrate strong zero-shot performance, but often perpetuate social stereotypes in person-centric queries, yielding skewed demographic distributions. Current debiasing methods apply uniform bias corrections across all input queries regardless of their bias sensitivity, creating a fundamental fairness--utility trade-off. Strong debiasing distorts semantically meaningful information in bias-insensitive queries, while weak debiasing fails to mitigate stereotypes in bias-sensitive ones. This one-size-fits-all approach hampers simultaneously achieving high utility on bias-insensitive queries and fairness on bias-sensitive queries. We introduce Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning-based test-time adaptation framework that selectively applies debiasing based on input sensitivity. RG-TTA adaptively triggers fairness regularization based on the bias sensitivity of each input during test-time policy adaptation, while focusing exclusively on optimizing cross-modal alignment for bias-insensitive inputs. Experiments on fairness benchmarks (e.g., FairFace, UTKFace) demonstrate substantial bias reduction while simultaneously improving zero-shot utility, resolving the trade-off of uniform debiasing.

Problem

• debiasing
• fairness
• vision-language models
• bias sensitivity
• utility trade-off

Innovation

• reward gating
• test-time adaptation
• selective debiasing
• vision-language models
• fairness-utility trade-off