Differentiable Zero-One Loss via Hypersimplex Projections

2026-02-26

AI Summary

This work addresses the challenge of optimizing the zero-one loss—a gold standard for classification performance that is non-differentiable and thus incompatible with gradient-based learning. The authors propose a novel differentiable approximation by introducing a smooth, order-preserving projection onto an \(n,k\)-dimensional hypersimplex, yielding a new operator termed Soft-Binary-Argmax. This operator is seamlessly integrated into an end-to-end learning framework, enabling, for the first time, direct differentiable approximation of the zero-one loss. The approach leverages the geometric constraints of the hypersimplex to enhance structural consistency and incorporates efficient Jacobian computation with constrained optimization. Empirical results demonstrate that the method significantly improves generalization in large-batch training scenarios, effectively narrowing the accuracy gap typically observed between large- and small-batch regimes.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in machine learning have emphasized the integration of structured optimization components into end-to-end differentiable models, enabling richer inductive biases and tighter alignment with task-specific objectives. In this work, we introduce a novel differentiable approximation to the zero-one loss-long considered the gold standard for classification performance, yet incompatible with gradient-based optimization due to its non-differentiability. Our method constructs a smooth, order-preserving projection onto the n,k-dimensional hypersimplex through a constrained optimization framework, leading to a new operator we term Soft-Binary-Argmax. After deriving its mathematical properties, we show how its Jacobian can be efficiently computed and integrated into binary and multiclass learning systems. Empirically, our approach achieves significant improvements in generalization under large-batch training by imposing geometric consistency constraints on the output logits, thereby narrowing the performance gap traditionally observed in large-batch training.

Problem

• zero-one loss
• differentiable approximation
• gradient-based optimization
• classification
• non-differentiability

Innovation

• differentiable zero-one loss
• hypersimplex projection
• Soft-Binary-Argmax
• structured optimization
• large-batch training