Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classification

2026-06-19

AI Summary

This work addresses the hierarchical inconsistency problem in multi-level visual classification, where fine-grained predictions often conflict with their parent categories. To mitigate this issue, the authors propose a hierarchy-constrained contrastive learning mechanism that performs contrastive optimization exclusively within the same semantic level, thereby eliminating interference from cross-level false negatives. Additionally, a group-balanced optimization strategy is introduced to ensure adequate training across all hierarchy levels. Built upon the BioCLIP framework, the method jointly optimizes representations in both Euclidean and hyperbolic spaces, significantly improving hierarchical consistency and classification performance. Evaluated on benchmarks including iNaturalist 2021, the approach achieves a 30.47% average improvement in cross-level accuracy over baseline methods and demonstrates notably enhanced consistency under zero-shot settings.

Abstract

Multimodal contrastive learning has enabled zero-shot visual classification by aligning images with textual categories. However, in hierarchically structured label spaces, existing methods often produce predictions that are inconsistent across taxonomic levels. For example, a model may predict a fine-grained category whose parent category contradicts its simultaneously predicted higher-level label. By analysis, the issue originates from false negative labels when contrastive comparison involves multiple taxonomic levels. To this end, we propose to restrict contrastive comparisons to categories within the same taxonomic level. In addition, we adopt a group-balanced design, ensuring each taxonomic level receives adequate optimization. As a result, the proposed framework improves both hierarchical consistency and classification accuracy from coarse to fine granularity. We train our model with TreeOfLife-10M based on BioCLIP and evaluate it across multiple hierarchical classification benchmarks, where the model demonstrates significantly improved hierarchical consistency in both Euclidean and hyperbolic spaces. Notably, on iNaturalist 2021 (iNat21), our method improves average accuracy across levels by 30.47% over the baseline, highlighting its effectiveness for hierarchical zero-shot classification.

Problem

• hierarchical classification
• zero-shot learning
• contrastive learning
• label inconsistency
• fine-grained vision classification

Innovation

• hierarchical contrastive learning
• level-restricted comparison
• zero-shot classification
• taxonomic consistency
• group-balanced optimization