FairNVT: Improving Fairness via Noise Injection in Vision Transformers

2026-04-17

AI Summary

This work addresses fairness issues at both representation and prediction levels in pretrained Vision Transformers (ViTs) by proposing a lightweight debiasing framework that jointly optimizes these two aspects without compromising task accuracy. The method injects calibrated Gaussian noise into sensitive embeddings, fuses them with task representations, and incorporates lightweight adapters, orthogonality constraints, and fairness-aware regularization to achieve efficient debiasing. Notably, this framework is the first to simultaneously handle representation and prediction fairness in a unified manner and is compatible with various pretrained ViT architectures. Evaluated on three vision-language datasets, it significantly reduces the success rate of sensitive attribute inference attacks while improving demographic parity and equalized odds, all while maintaining strong downstream task performance.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

This paper presents FairNVT, a lightweight debiasing framework for pretrained transformer-based encoders that improves both representation and prediction level fairness while preserving task accuracy. Unlike many existing debiasing approaches that address these notions separately, we argue they are inherently connected: suppressing sensitive information at the representation level can facilitate fairer predictions. Our approach learns task-relevant and sensitive embeddings via lightweight adapters, applies calibrated Gaussian noise to the sensitive embedding, and fuses it with the task representation. Together with orthogonality constraints and fairness regularization, these components jointly reduce sensitive-attribute leakage in the learned embeddings and encourage fairer downstream predictions. The framework is compatible with a wide range of pretrained transformer encoders. Across three datasets spanning vision and language, FairNVT reduces sensitive-attribute attacker accuracy, improves demographic-parity and equalized-odds metrics, and maintains high task performance.

Problem

• fairness
• vision transformers
• debiasing
• representation learning
• sensitive attributes

Innovation

• noise injection
• fairness
• vision transformers
• debiasing
• representation learning