Learning from Hard Labels with Additional Supervision on Non-Hard-Labeled Classes

2025-07-24

AI Summary

To address the insufficiency of hard-label supervision in few-shot classification, this paper proposes leveraging the distributional structure—rather than mere confidence scores—of non-ground-truth classes as auxiliary supervisory signals. Specifically, soft labels are constructed via affine combinations and jointly optimized over both direction and step size within the probability simplex. Theoretically, we establish for the first time how such distributional information fundamentally influences the convergence rate and asymptotic value of the generalization error bound. Mechanistically, we reveal its complementary role with mixing coefficients in soft-label optimization. Extensive experiments demonstrate that the proposed method significantly enhances generalization: it achieves average accuracy improvements of 2.3–5.7 percentage points across multiple standard few-shot benchmarks.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In scenarios where training data is limited due to observation costs or data scarcity, enriching the label information associated with each instance becomes crucial for building high-accuracy classification models. In such contexts, it is often feasible to obtain not only hard labels but also {it additional supervision}, such as the confidences for the hard labels. This setting naturally raises fundamental questions: {it What kinds of additional supervision are intrinsically beneficial?} And {it how do they contribute to improved generalization performance?} To address these questions, we propose a theoretical framework that treats both hard labels and additional supervision as probability distributions, and constructs soft labels through their affine combination. Our theoretical analysis reveals that the essential component of additional supervision is not the confidence score of the assigned hard label, but rather the information of the distribution over the non-hard-labeled classes. Moreover, we demonstrate that the additional supervision and the mixing coefficient contribute to the refinement of soft labels in complementary roles. Intuitively, in the probability simplex, the additional supervision determines the direction in which the deterministic distribution representing the hard label should be adjusted toward the true label distribution, while the mixing coefficient controls the step size along that direction. Through generalization error analysis, we theoretically characterize how the additional supervision and its mixing coefficient affect both the convergence rate and asymptotic value of the error bound. Finally, we experimentally demonstrate that, based on our theory, designing additional supervision can lead to improved classification accuracy, even when utilized in a simple manner.

Problem

• Enhancing classification accuracy with limited training data
• Identifying beneficial types of additional supervision
• Theorizing how additional supervision improves generalization

Innovation

• Combines hard and soft labels via affine combination
• Uses non-hard-labeled class distribution information
• Determines optimal mixing coefficient for supervision