BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models

2025-09-28

AI Summary

Sentiment analysis (SA) models often harbor implicit societal biases, yet existing bias detection approaches rely on manually curated test sentence sets—costly to construct and limited in coverage and naturalness. Method: We propose a controllable large language model (LLM)-based generation framework that models bias dimensions and employs sentence perturbation control to automatically synthesize low-resource, high-naturalness bias test cases. Unlike conventional prompting methods, our framework enables cross-bias-type generalization and enhances linguistic diversity and test coverage. Contribution/Results: Experiments demonstrate that our method significantly outperforms baselines in bias detection rate, syntactic and semantic richness, and domain generalization. It provides a scalable, interpretable, and automated solution for fairness evaluation of SA models, advancing practical bias auditing in real-world NLP systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Sentiment Analysis (SA) models harbor inherent social biases that can be harmful in real-world applications. These biases are identified by examining the output of SA models for sentences that only vary in the identity groups of the subjects. Constructing natural, linguistically rich, relevant, and diverse sets of sentences that provide sufficient coverage over the domain is expensive, especially when addressing a wide range of biases: it requires domain experts and/or crowd-sourcing. In this paper, we present a novel bias testing framework, BTC-SAM, which generates high-quality test cases for bias testing in SA models with minimal specification using Large Language Models (LLMs) for the controllable generation of test sentences. Our experiments show that relying on LLMs can provide high linguistic variation and diversity in the test sentences, thereby offering better test coverage compared to base prompting methods even for previously unseen biases.

Problem

• Detecting social biases in sentiment analysis models
• Generating diverse test cases with linguistic variation
• Reducing reliance on domain experts for bias testing

Innovation

• LLMs generate bias test cases automatically
• Framework creates diverse sentences with minimal input
• LLMs provide better coverage than base prompting