Instance-Level Costs for Nuanced Classifier Evaluation

2026-05-04

AI Summary

Standard classification evaluation treats all errors equally, overlooking the fact that misclassifying clear instances incurs substantially higher costs than misclassifying ambiguous ones in real-world scenarios. To address this limitation, this work proposes the Normalized Excess Cost (NEC) metric and establishes the first instance-level framework for evaluating misclassification costs. It further introduces practical methods to derive instance-specific costs using annotation voting, model confidence, and other signals. Building on this framework, the authors integrate cost-sensitive training via loss weighting, sampling strategies, and regression-based approaches, demonstrating effectiveness across text, image, and tabular data. Experiments show that while models may exhibit a 5% error rate, their NEC can be as low as 1.8%; however, cost-sensitive training yields significant gains only when misclassification costs are predictable from input features, limiting its benefits in many real-world settings.

Abstract

Standard classification treats all errors equally, but in content moderation, medical screening, and safety-critical applications, mistakes on clear-cut cases are far more costly than errors on ambiguous ones. We propose normalized excess cost (NEC), a metric that weights classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate -- models with 5\% error rate can achieve 1.8\% NEC -- revealing that most mistakes concentrate on ambiguous, low-cost examples. However, incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits: improvements appear only when costs are predictable from input features, as in our synthetic control, while real-world datasets show mixed or negligible gains. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even when cost-sensitive training offers limited benefit.

Problem

• cost-sensitive evaluation
• instance-level costs
• classification errors
• ambiguous examples
• misclassification cost

Innovation

• normalized excess cost
• instance-level costs
• cost-sensitive evaluation
• classification metrics
• ambiguous examples