PRSM: A Measure to Evaluate CLIP's Robustness Against Paraphrases

2025-11-14

AI Summary

This work investigates the robustness of CLIP models to paraphrasing—i.e., semantically preserving rewordings—in socially sensitive contexts, exposing critical fairness risks in real-world deployment. We propose the Paraphrase Ranking Stability Measure (PRSM), the first systematic metric quantifying CLIP’s consistency in ranking image–text alignments across diverse automated and human-generated paraphrases. Leveraging the Social Counterfactuals dataset, we conduct bias-aware correlation analysis. Experiments reveal pronounced sensitivity disparities across paraphrase types and uncover a stable, consistent robustness gap in gender-related queries: CLIP exhibits higher response stability for male-associated prompts but significantly greater vulnerability to paraphrase perturbations for female-associated ones. Our framework establishes a novel, reproducible paradigm for evaluating linguistic robustness in multimodal foundation models, providing actionable metrics for fairness auditing and trustworthy deployment.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Contrastive Language-Image Pre-training (CLIP) is a widely used multimodal model that aligns text and image representations through large-scale training. While it performs strongly on zero-shot and few-shot tasks, its robustness to linguistic variation, particularly paraphrasing, remains underexplored. Paraphrase robustness is essential for reliable deployment, especially in socially sensitive contexts where inconsistent representations can amplify demographic biases. In this paper, we introduce the Paraphrase Ranking Stability Metric (PRSM), a novel measure for quantifying CLIP's sensitivity to paraphrased queries. Using the Social Counterfactuals dataset, a benchmark designed to reveal social and demographic biases, we empirically assess CLIP's stability under paraphrastic variation, examine the interaction between paraphrase robustness and gender, and discuss implications for fairness and equitable deployment of multimodal systems. Our analysis reveals that robustness varies across paraphrasing strategies, with subtle yet consistent differences observed between male- and female-associated queries.

Problem

• Evaluating CLIP's robustness against linguistic paraphrasing variations
• Measuring sensitivity to paraphrased queries using PRSM metric
• Assessing interaction between paraphrase robustness and demographic biases

Innovation

• Introducing PRSM metric for CLIP paraphrase robustness
• Using Social Counterfactuals dataset for bias assessment
• Evaluating robustness variation across paraphrasing strategies