Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

2026-07-02

AI Summary

This work addresses the inconsistent performance of large multimodal models in hierarchical visual recognition due to their lack of explicit taxonomic knowledge. To this end, the authors propose HiR², a plug-and-play hierarchical representation regularization method that introduces, for the first time, a dual-objective regularization mechanism into multimodal models. Specifically, it constructs a semantics-aware visual hierarchy via hyperbolic entailment cones embedded in Lorentz space to preserve radial hierarchical structure, while simultaneously imposing an angular separation loss on the unit hypersphere to enhance the discriminability of semantically similar embeddings. By effectively fusing intermediate linguistic features with semantics-guided visual representations, HiR² significantly improves hierarchical recognition consistency across various state-of-the-art multimodal architectures and fine-tuning paradigms, while better capturing taxonomic structures across tasks.

Abstract

Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization ($HiR^2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that $HiR^2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.

Problem

• taxonomic knowledge
• hierarchical visual recognition
• large multimodal models
• hierarchical consistency
• semantic relationships

Innovation

• Hierarchical Representation Regularization
• Taxonomic Entailment
• Hyperbolic Geometry
• Multimodal Learning
• Visual Semantic Hierarchy