Debiasing Large Language Models in Thai Political Stance Detection via Counterfactual Calibration

2025-09-26

AI Summary

Large language models (LLMs) exhibit systematic biases—such as sentiment leakage and entity preference—in Thai political stance detection, particularly under low-resource and culturally complex conditions. Method: We propose ThaiFACTUAL, a lightweight, model-agnostic calibration framework that mitigates bias without fine-tuning. It employs counterfactual data augmentation and rationale-driven supervision to disentangle sentiment expression from genuine political stance. Contribution/Results: We introduce the first high-quality, multidimensionally annotated Thai political stance dataset and design culture-sensitive calibration strategies. Experiments demonstrate that ThaiFACTUAL significantly reduces multiple bias types, improves zero-shot performance and cross-model fairness across diverse LLMs, and generalizes effectively across varied political entities and events.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Calibration & Uncertainty Quantification
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Political stance detection in low-resource and culturally complex settings poses a critical challenge for large language models (LLMs). In the Thai political landscape - marked by indirect language, polarized figures, and entangled sentiment and stance - LLMs often display systematic biases such as sentiment leakage and favoritism toward entities. These biases undermine fairness and reliability. We present ThaiFACTUAL, a lightweight, model-agnostic calibration framework that mitigates political bias without requiring fine-tuning. ThaiFACTUAL uses counterfactual data augmentation and rationale-based supervision to disentangle sentiment from stance and reduce bias. We also release the first high-quality Thai political stance dataset, annotated with stance, sentiment, rationales, and bias markers across diverse entities and events. Experimental results show that ThaiFACTUAL significantly reduces spurious correlations, enhances zero-shot generalization, and improves fairness across multiple LLMs. This work highlights the importance of culturally grounded debiasing techniques for underrepresented languages.

Problem

• Debiasing LLMs in Thai political stance detection using counterfactual calibration
• Addressing sentiment leakage and entity favoritism in low-resource settings
• Disentangling sentiment from stance in culturally complex political landscapes

Innovation

• Counterfactual data augmentation for bias mitigation
• Rationale-based supervision disentangles sentiment from stance
• Lightweight model-agnostic calibration without fine-tuning