Hyperbolic Residual Quantization: Discrete Representations for Data with Latent Hierarchies

2025-05-18

AI Summary

Existing residual quantization (RQ) methods suffer from geometric mismatch when modeling hierarchical data—such as knowledge graphs and biological taxonomies—in Euclidean space. To address this, we propose Hyperbolic Residual Quantization (HRQ), the first RQ framework incorporating hyperbolic geometry. HRQ introduces hierarchical inductive bias via hyperbolic embedding, hyperbolic distance metrics, and hyperbolic residual operations, enabling natural modeling of latent hierarchical branches through multi-level hyperbolic codebooks. Evaluated on WordNet hierarchy modeling and unsupervised hierarchical discovery tasks, HRQ yields discrete representations that improve downstream task performance by up to 20% over Euclidean RQ. This advancement overcomes a fundamental limitation of conventional RQ methods in hierarchical representation learning, demonstrating the superiority of hyperbolic geometry for capturing tree-like structural priors.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Hierarchical data arise in countless domains, from biological taxonomies and organizational charts to legal codes and knowledge graphs. Residual Quantization (RQ) is widely used to generate discrete, multitoken representations for such data by iteratively quantizing residuals in a multilevel codebook. However, its reliance on Euclidean geometry can introduce fundamental mismatches that hinder modeling of hierarchical branching, necessary for faithful representation of hierarchical data. In this work, we propose Hyperbolic Residual Quantization (HRQ), which embeds data natively in a hyperbolic manifold and performs residual quantization using hyperbolic operations and distance metrics. By adapting the embedding network, residual computation, and distance metric to hyperbolic geometry, HRQ imparts an inductive bias that aligns naturally with hierarchical branching. We claim that HRQ in comparison to RQ can generate more useful for downstream tasks discrete hierarchical representations for data with latent hierarchies. We evaluate HRQ on two tasks: supervised hierarchy modeling using WordNet hypernym trees, where the model is supervised to learn the latent hierarchy - and hierarchy discovery, where, while latent hierarchy exists in the data, the model is not directly trained or evaluated on a task related to the hierarchy. Across both scenarios, HRQ hierarchical tokens yield better performance on downstream tasks compared to Euclidean RQ with gains of up to $20%$ for the hierarchy modeling task. Our results demonstrate that integrating hyperbolic geometry into discrete representation learning substantially enhances the ability to capture latent hierarchies.

Problem

• Improving discrete hierarchical representations using hyperbolic geometry
• Addressing Euclidean geometry mismatch in hierarchical data modeling
• Enhancing latent hierarchy capture in multitoken representation learning

Innovation

• Hyperbolic embedding for hierarchical data representation
• Hyperbolic operations and distance metrics for quantization
• Improved discrete hierarchical representations for downstream tasks