Position Bias in Ordinal Classification: A Systematic Evaluation

2026-08-09

AI Summary

This study systematically investigates position bias in large language models (LLMs) for ordinal classification, where predictions are unduly influenced by semantically irrelevant factors such as label order and example placement in prompts, compromising system stability. Through controlled experiments across five datasets, the authors evaluate ten state-of-the-art LLMs, analyzing eight factors spanning prompt design, task formulation, and model architecture, while comparing pointwise, pairwise, and listwise reasoning paradigms alongside multiple debiasing strategies. Findings reveal that all models exhibit sensitivity to three types of position bias; reducing label cardinality concurrently improves both accuracy and stability; listwise methods generally perform best but show inconsistent generalization; and existing debiasing techniques lack universal efficacy. Crucially, predictive performance and stability often diverge, with robustness depending on the entire system configuration rather than the model alone.

Abstract

Large language models are increasingly used for ordinal classification, yet semantically equivalent changes to prompt organization can alter their predictions. We conduct systematic experiments to characterize positional bias from label order, demonstration order, and demonstration placement. First, we apply the three probes to ten frontier LLMs on a common ordinal-classification task; every model is sensitive to all three positional sources, showing that the problem is pervasive. Second, we vary eight prompt-, task-, and model-level factors across five datasets; accuracy and stability are often misaligned, and only lower scale cardinality consistently improves both. Third, we compare pointwise, pairwise, and listwise inference, alternative aggregation and debiasing methods, and joint configurations; the tested corrections do not provide a reliable remedy, while a comparison-based listwise formulation offers the best balance but transfers unevenly across models and bias sources. These findings show that positional robustness depends on the full system configuration rather than the model alone. Ordinal-classification systems should therefore be selected jointly for predictive performance and stability.

Problem

• Position Bias
• Ordinal Classification
• Large Language Models
• Prompt Engineering
• Robustness

Innovation

• positional bias
• ordinal classification
• prompt engineering
• large language models
• systematic evaluation